In sci-fi future movies like 5th Element they comically make AI technology seem kinda dumb - or at least like you can trick it.

You can make permissions decide which model is used, and only train that on the data that permission allows (someone should build this - permissioned RAG), that solves leaks, but decision-making within a model, or behavior exfiltration comparable to viewing backend source code still seems possible.

It’s basically like client/server security: You can’t “trust the model” in the same way you can’t “trust the client” in a backend/frontend setup.

When used as effectively a point-of-sale, AI seems more hackable than a vending machine - as a boss or assistant even more so.

Exactly. This is how I design agents as well. I essentially treat the agent as the web/mobile app, cli tool os API library not as part of my backend even though there is where it runs. The backend doesn't treat the agent in any special way. It simply gates all action based on the permissions of the user/guest that is using the agent.