In principle they could output meaningful such language if they were capable of metacognition, which so far doesn't seem to be a goal of AI developers (and rightfully so, since they achieved so many miracles bypassing it).

As far as I know we don't know much about metacognition in LLMs, though? Not sure

[dead]