I'm not a lawyer so no comment on the actual legality, but it feels like the issue in your example is more akin to negligence than it is ill intent. That an LLM can misrepresent / hallucinate things is foundational to the technology.

I feel it's somewhat similar to using TurboTax or other online services to file your taxes. For those not in the US, the tax forms have a special field to fill in if the form was prepared by a paid preparer (licensed, I guess); TurboTax and other online services leave that field blank so you, as a taxpayer, are responsible for all possible mistakes that you didn't see in the forms you signed and sent.

I feel it's very similar to using an agent; the main difference is that we kind of trust that TurboTax won't hallucinate in your forms, while the agent surely will.

A great number of things that people in chat rooms “can definitely see” end up sounding awfully ridiculous in court.

> I can definitely see someone being able to mount a good criminal defence case that they aren’t responsible for misrepresentations that an LLM agent makes on, for example, a loan application, if they sincerely intended to use the agent for non-fraudulent purposes. There won’t be intent, so they aren’t responsible outside of strict liability.

i mean, no? how would that be a good defence? "yes your honour, i lied on a loan application and committed fraud, but it was a mistake! i promise!" - that's... yeah. fine. it's not exactly unique.

regardless, it's _you_ making the loan application. not the agent. your failure to check it is on you.

Your sign the document affirming that the information is accurate.

There's not really enough info in the article to decide. I'd have no problem with him going to court and having to show to a jury that he used the bot in a way that no reasonable person would have expected to result in the website being hacked. If that was found to be the case, the company who made the chatbot would be responsible for the harms caused to the gym, and the user, including all of the legal costs he incurred by having to defend himself.

As for the people who couldn't attend the pilates class because of the hack, it's the gym who should be held accountable for that because they designed their system in such an insecure and negligent way that it failed to protect their data and resulted in them losing their place.