Discerning the limits is the most important thing of all, and we seem very eager to obfuscate it for LLMs.

We so desperately want something we can sell as AGI or at least magic that the boundaries on the tools are few, far-between, and mostly based on legal needs "don't generate nudes of celebrities who can sue us" rather than grasped technical limits.

The more complex and sophisticated the query, the harder it will be to double-check and make sure you're still on the rails. So it's the responsibility of the people offering the tools to understand and define their limits before customers unknowningly push their legal-assistant LLMs into full Sovereign Citizen mode.