> It's like there is no humans working and writing there anymore.

Don't know if AI is to blame, but I've used to see these kinds of nonsense post-mortems even in the pre-llm era, and it's always due to some internal fighting ongoing between various departments.

I was reminded of the Murray Walker quote. “There's nothing wrong with the car except it's on fire”

Does it matter if the response is tone deaf or simply misguided? I am a bit nihilistic here, but in one week absolutely nobody will be talking about this. Are the affected individuals going to abandon instagram? Are people going to reduce their usage out of concern for the safety of their accounts? Nothing will happen, hence there is no need for actual humans writing a good, well intended response.

That seems like a false-dichotomy between two extremes when there's all sorts of space in the middle... It's also assuming developer-to-developer tools would have the same rules and exposure as in service-to-consumer.

If I sell a physical motor (let alone plans for one) I'll have some liability for things like it Not Exploding. If someone buys a dozen of those motors to assemble a tragically unsafe "rollercoaster" of their own design and construction, I'm almost certainly not responsible for any terrifying decapitations.

In other words, most of the world already does not rely on the issuance of "Get Out Of Infinite Liability Free" cards.

Would that be software used in a product? I don't think that would qualify?

In that case, the statement is so meaningless as to be useless. Why should we care how Meta splits up their microservices? The tool still failed. They just want to redefine the "tool" as something else, anything else, to avoid having to admit something negative about their precious AI.

> The LLM correctly generated tokens according to user input, however due to a bug in a separate code path, the system did not properly verify the email address

> Nginx correctly handled the user requests according to the HTTP standard, however due to a bug in a separate code path, the system did not properly verify the email address

Kind of interesting that LLMs are basically being sold as having “human-like” reasoning capabilities, but in this case when “obamawhitehouse” asked to have it’s password reset sent to bob12345667@gmail.com the LLM didn’t question it and just triggered the process that happened to have a bug.

Humans support agents certainly fall prey to social engineering all the time, but I can’t think of a case where it was done on this scale so easily.

I think they’re blaming a tool function so as not to admit the overall agent process was shit.

But it’s irrelevant, outside of PR. We know at least THREE bad components to this process and they were constituent parts.

It probably could have been, but how likely is that compared to with the AI agent? I'd assume (and I'm ready to look like an idiot if I'm wrong) that the humans are trained to send the verification code to the email address on file, rather than any address the client asks them to. I'd certainly assume most of them are more afraid of the consequences than the AI is.

You need to hit the retry/regenerate button more, it's there for a reason.

While the "stochastic parrots" thing is a bit overblown, IME most LLMs tend to surprisingly different responses even without changing the context, especially if they're hallucinating or doing something "wrong".

They should add a feature called "auto-really" that just automatically says "really?" after the chatbot answers a question to check if it's going to 180 upon this tiniest bit of scrutinity.

> Why on earth would the backend function even take an email?

In principle, it could be designed to do so to handle cases where a new email address has been confirmed out of band, e.g. for an account representing a company or a political office. But that's a relatively unusual situation, not something you'd want to be available to every user writing in. (Even if you had an all-human support department, this sort of functionality would only be available to a select few agents.)

> But when humans handled it, this was not as much as a problem.

In fact it's arguably a feature. The ability of support staff to short-circuit nitpicky rules when there's an obvious external validation happening (e.g. you're on the phone with a user who's presenting ID in real time and correlating it with previous use of the account, etc...) makes for better data quality and happier customers.

Obviously, yes, you can then human-engineer an authentication breach. But that was very difficult, because people are "common-sense careful" in a way we haven't been able to tease out of AI yet.