This exploit has essentially nothing to do with AI and everything to do with a terribly designed account recovery flow.

This exact same flow could have been (and may have been; I don’t know how much the chatbot here actually does) statically coded.

The AI part does seem relevant because it enabled incredibly low-effort “social” engineering.

For what it’s worth I don’t think you can call this social engineering since there was no human on the other end, even though it appears similar.

The question is, if there were actual human support agents, would they have built additional safeguards to prevent social engineering in this manner?

My impression is that AI didn't replace static code in this place; it replaced a person, who (hopefully) would have been suspicious about sending an account recovery code for e.g. "obamawhitehouse" to e.g. "bscurtu.alfamm.ro@gmail.com"

This is not true. Well, it kinda is, but nobody will be stupid enough to hand-code an account recovery where you get to type any email address.

The reason it worked there is that the designers of the system didn't anticipate that the AI will agree to accept any email (maybe they even put guardrails against it in the system prompt, we don't know). It's more like social engineering than bad-security-code, except that like the sibling comment said an actual human will probably not approve that.

> This exact same flow could have been…statically coded.

But had never been until it was wrapped in a chatbot. It’s just about unheard of for a major site in the modern era, isn’t it? I think the AI factor is essentially essential. All but.

The reason all these meticulously designed flows have been done away with is because some manager believes that AI is omniscient and can just replace it all.

Like, flagging VPN endpoints is bread and butter for this kind of thing and must already exist. But it's been bypassed

An email address is making its way from a publicly available LLM prompt input to a sensitive email's recipient address. That's the problem I'm highlighting.

That may be but I think it's fair to say that AI is more suggestible than people.

I agree with your point, mostly.

Until I remember seeing someone saying "MCP is dead, we just give agents command line access now". Then I start to think that looking at this in the context of ai is helpful.

its AI-INCOMPETENCE. the blame is coming from the top.

dontake excuses for the greedy

Drowning has essentially nothing to do with water and everything to do with a terribly designed ability to get air into your lungs.

If you'd do a retrospective and ignore how AI has shaped expectations and a company's culture to allow this to pass through into production, you'd be complicit/perpetuating what led to this debacle in the first place.

It's not the end of the world, and water isn't going anywhere, but saying AI has essentially nothing to do with it is just a bad take.

This sounds like it was “designed” by an actual idiot. Maybe vibe coded on a Saturday.

Nobody would handcraft a password reset flow that ignores the users' email and 2fa settings lol

Also I've used Meta's old password recovery system. It's not possible to do this in that version. The chatbot is what makes this possible.

Account recovery (forgot password) doesn't actually require human or Ai in the loop?

I mean this particular auth flow has been a well-known pattern, even before Ai came along.

I guess the only way they got away with this is due to the Ai in the loop. They kind of social (artificial) engineered the Ai, which prolly overlooked the well-known password recovery pattern.

Vibe coded?