Yep, thinking LLMs are just a tool just isn't a good mental model of what's occurring.

Of course HN has a pretty large number of people that say things like "But I want the nuclear weapon building machine to do whatever I want without asking questions, and there are no possible wide scale ramifications that will happen because of this".

These AI models are something new and different and we need to change our expectations of them over conventional software.