Often times I'll say something like:

"Can we make the change to change the button color from red to blue?"

Literally, this is a yes or no question. But the AI will interpret this as me _wanting_ to complete that task and will go ahead and do it for me. And they'll be correct--I _do_ want the task completed! But that's not what I communicated when I literally wrote down my thoughts into a written sentence.

I wonder what the second order effects are of AIs not taking us literally is. Maybe this link??

Such miscommunication (varying levels of taking it literally) is also common with autistic and allistic people speaking with each other

I don't find that an unreasonable interpretation. Absent that paragraph of explained thought process, I could very well read it the agent's way. That's not a defect in the agent, that's linguistic ambiguity.

It's funny because I interpret it the opposite way you do. If someone asked me that question, I'd absolutely assume they want it changed and do it.

If you work with codex a lot you’ll find it is good at taking you literally, and that that is almost never what you want.

I mean humans communicate the same way. We don't interpret the words literally and neither does the LLM. We think about what one is trying to communicate to the other.

For example If you ask someone "can you tell me what time it is?", the literal answer is either "yes"/"no". If you ask an LLM that question it will tell you the time, because it understands that the user wants to know the time.