One inherent limitation of current LLM/AI is that they are primarily trained on abstracted data that focuses primarily on mimicking our logical and reasoning prefrontal cortex portion of the mind. However most humans make decisions based on activity in the limbic regions of the brain which are essentially emotional and intuition based. So we often will do something before we actually know why we did it, however to maintain a sense of self and sanity we will then use our prefrontal cortex to create a cohesive narrative on why we do what we do (despite it often being inaccurate).

In a nutshell we are mimicking neural activity in a certain region based on certain abstracted data which is quite removed from how we as humans process reality.

In the same sense that split brain patients [1] will make up a reasonable explanation for what the other half did.

And why witnesses are preferably interviewed very shortly after they witnessed a crime. Before their brains start to 'fill in the blanks'

1: https://en.wikipedia.org/wiki/Split-brain