I agree with everything you say except this:

> You can make any modern LLM explain its reasoning

You can make any modern LLM create a plausible, self-consistent explanation that looks like reasoning, but it's not "the reasoning it used to arrive at that answer".

▲

kentonv 5 hours ago | parent | next [-]

Tangent: This is often true of humans as well.

We often make a decision based on a gut feeling, and then backfill a logical reason supporting our feeling, without even realizing we're doing it -- rationalization.

	▲	angry_octet 2 hours ago \| parent \| next [-]
		And we all know some people that rationalise poor choices and misbehavior, hide their mistakes, etc, to an unacceptable degree. Sometimes the individual knows they are rationalising but continues anyway, other times they seem incapable of seeing that. When you ask people who are rationalising poor behaviour about the scenario, but it is someone else doing it, they may arrive at a better answer. Can we use multiple LLMs to achieve self criticism and critical thinking?
	▲	glenstein an hour ago \| parent \| prev [-]
		Tangent on the tangent: I think that's true in a minority of cases and in a majority of AI cases. Though in principle I think it should be possible for an LLM to have access to and faithfully represent its own reasoning.

▲

dist-epoch 6 hours ago | parent | prev [-]

This was beautifully shown by asking a model to explain how it added two numbers together (something like 45+21), and it told a plausible story, when in fact they showed it was some rotation on a helix living in some internal manifold.

Like asking a human "how did you catch that fast ball coming at you?"

	▲	markasoftware 3 hours ago \| parent [-]
		It could be that the rotation in the helix manifold whatever is a low level representation of the logical steps (carry the 2, add the next column,...) it's describing. The point stands that the explanation it generates doesn't necessarily in all cases reflect what it "actually did" but your counterexample doesn't hold.