| ▲ | pierrekin 4 hours ago |
| I agree that the model can help troubleshoot and debug itself. I argue that the model has no access to its thoughts at the time. Split brain experiments notwithstanding I believe that I can remember what my faulty assumptions were when I did something. If you ask a model “why did you do that” it is literally not the same “brain instance” anymore and it can only create reasons retroactively based on whatever context it recorded (chain of thought for example). |
|
| ▲ | XenophileJKO 3 hours ago | parent | next [-] |
| Anthropic's introspection experiments have seemed to show that your argument is falsifiable. https://www.anthropic.com/research/introspection |
| |
| ▲ | sumeno 2 hours ago | parent [-] | | > In fact, most of the time models fail to demonstrate introspection—they’re either unaware of their internal states or unable to report on them coherently. You got the wrong takeaway from your link. | | |
| ▲ | XenophileJKO 2 hours ago | parent [-] | | The parent said: "I argue that the model has no access to its thoughts at the time." This is falsified by that study, showing that on the frontier models generalized introspection does exist. It isn't consistent, but is is provable. "no access" vs. "limited access" | | |
| ▲ | sumeno 2 hours ago | parent | next [-] | | There is no way for a user to know whether the LLM has introspection in a given case or not, and given that the answer is almost always no it is much better for everyone to assume that they do not have introspection. You cannot trust that the model has introspection so for all intents and purposes for the end user it doesn't. | |
| ▲ | dwheeler 2 hours ago | parent | prev [-] | | I would say "limited and unreliable access". What it says is the cause might be the cause, but it's not on any way certain. |
|
|
|
|
| ▲ | fragmede 4 hours ago | parent | prev | next [-] |
| Claude code and codex both hide the Chain of Thought (CoT) but it's just words inside a set of <thinking> tags </thinking> and the agent within the same session has access to that plaintext. |
| |
| ▲ | fc417fc802 3 hours ago | parent [-] | | Those are just words inside arbitrary tags, they aren't actually thoughts. Think of it as asking the model to role play a human narrating his internal thought process. The exercise improves performance and can aid in human understanding of the final output but it isn't real. | | |
| ▲ | antonvs 3 hours ago | parent [-] | | Why do you believe that humans have access to an “internal thought process”? I.e. what do you think is different about an agent’s narration of a thought process vs. a human’s? I suspect you’re making assumptions that don’t hold up to scrutiny. | | |
| ▲ | fc417fc802 2 hours ago | parent | next [-] | | I made no such claim and I don't understand what direct relevance you believe the human thought process has to the issue at hand. You appear to be defaulting to the assumption that LLMs and humans have comparable thought processes. I don't think it's on me to provide evidence to the contrary but rather on you to provide evidence for such a seemingly extraordinary position. For an example of a difference, consider that inserting arbitrary placeholder tokens into the output stream improves the quality of the final result. I don't know about you but if I simply repeat "banana banana banana" to myself my output quality doesn't magically increase. | |
| ▲ | 2 hours ago | parent | prev [-] | | [deleted] |
|
|
|
|
| ▲ | jmalicki 4 hours ago | parent | prev [-] |
| It does have access to its thoughts. This is literally what thinking models do. They write out thoughts to a scratch pad (which you can see!) and use that as part of the prompt. |
| |
| ▲ | fc417fc802 4 hours ago | parent | next [-] | | It's important to be aware that while those "thoughts" can be a useful aid for human understanding they don't seem to reliably reflect what's going on under the hood. There are various academic papers on the matter or you can closely inspect the traces of a more logically oriented question for yourself and spot impossible inconsistencies. | |
| ▲ | mmoll 4 hours ago | parent | prev | next [-] | | It doesn’t mean that these “thoughts” influenced their final decision the way they would in humans. An LLM will tell you a lot of things it “considered” and its final output might still be completely independent of that. | | |
| ▲ | jmalicki 2 hours ago | parent [-] | | Its output quite literally is not independent, as the "thinking tokens" are attended to by the attention mechanism. |
| |
| ▲ | grey-area 4 hours ago | parent | prev | next [-] | | They do not in fact do that. The ‘thoughts’ are not a chain of logic. | |
| ▲ | sumeno 2 hours ago | parent | prev | next [-] | | You have a fundamental misunderstanding of what the model is doing. It's not your fault though, you're buying into the advertising of how it works | |
| ▲ | 4 hours ago | parent | prev [-] | | [deleted] |
|
