>> really don't understand how people given access to a pile of tools and data sources and unleash them on customers

It’s pretty simple. When a non-tech person sees faked demos of what it can do - it looks epic and everyone extrapolates results and thinks AI is that good.

Doubly so if the person deciding what gets implemented doesn't really get what their staff actually do.

LLMs ability to give convincing sounding answers is like catnip for service desk managers who have never actually been on the desk itself