The AI part does seem relevant because it enabled incredibly low-effort “social” engineering.

For what it’s worth I don’t think you can call this social engineering since there was no human on the other end, even though it appears similar.

The question is, if there were actual human support agents, would they have built additional safeguards to prevent social engineering in this manner?

My impression is that AI didn't replace static code in this place; it replaced a person, who (hopefully) would have been suspicious about sending an account recovery code for e.g. "obamawhitehouse" to e.g. "bscurtu.alfamm.ro@gmail.com"

This is not true. Well, it kinda is, but nobody will be stupid enough to hand-code an account recovery where you get to type any email address.

The reason it worked there is that the designers of the system didn't anticipate that the AI will agree to accept any email (maybe they even put guardrails against it in the system prompt, we don't know). It's more like social engineering than bad-security-code, except that like the sibling comment said an actual human will probably not approve that.

> This exact same flow could have been…statically coded.

But had never been until it was wrapped in a chatbot. It’s just about unheard of for a major site in the modern era, isn’t it? I think the AI factor is essentially essential. All but.

The reason all these meticulously designed flows have been done away with is because some manager believes that AI is omniscient and can just replace it all.

Like, flagging VPN endpoints is bread and butter for this kind of thing and must already exist. But it's been bypassed

An email address is making its way from a publicly available LLM prompt input to a sensitive email's recipient address. That's the problem I'm highlighting.

That may be but I think it's fair to say that AI is more suggestible than people.

I agree with your point, mostly.

Until I remember seeing someone saying "MCP is dead, we just give agents command line access now". Then I start to think that looking at this in the context of ai is helpful.

Drowning has essentially nothing to do with water and everything to do with a terribly designed ability to get air into your lungs.

If you'd do a retrospective and ignore how AI has shaped expectations and a company's culture to allow this to pass through into production, you'd be complicit/perpetuating what led to this debacle in the first place.

It's not the end of the world, and water isn't going anywhere, but saying AI has essentially nothing to do with it is just a bad take.

its AI-INCOMPETENCE. the blame is coming from the top.

dontake excuses for the greedy

This sounds like it was “designed” by an actual idiot. Maybe vibe coded on a Saturday.

Nobody would handcraft a password reset flow that ignores the users' email and 2fa settings lol

Also I've used Meta's old password recovery system. It's not possible to do this in that version. The chatbot is what makes this possible.

Account recovery (forgot password) doesn't actually require human or Ai in the loop?

I mean this particular auth flow has been a well-known pattern, even before Ai came along.

I guess the only way they got away with this is due to the Ai in the loop. They kind of social (artificial) engineered the Ai, which prolly overlooked the well-known password recovery pattern.

Vibe coded?

> but the engineers did not add a sufficiently thorough e2e test case to test the tool call against unrelated email verification attempts being provided to the tool call.

I'd go out on a limb to say the tests were likely AI generated. It's easy to miss a case like this one given that models like to generate a ton of test code that 'look' good at a glance but have subtle logic bugs that could potentially defeat the purpose of the test itself.

My own anecdata here, Claude generated a JUnit test with all the right setup, but missed a crucial assertion (there were very many other minor assertions) which made the test useless mostly.

Seems like the most plausible explanation. OTOH it feels like this is the sort of thing that might have been discovered/mitigated more quickly had there been a human in the loop.

The human vetting was that it was cheaper. Someone probably got promoted for it.

And zero accountability too. No one will be found and detected.

As a "young person" (under 30), my thoughts: There's a minority of us that do genuinely care, possibly more than most - so hiring someone from this minority would be helpful - but the vast majority of my peers don't care about privacy nor security. They often take this defeatist mindset of "my data is already out there, why should I care?", or prefer convenience over security. For example, "why should I switch to Signal if I have a public Instagram profile?" or "I can't remember all those passwords! I just use one for everything."

As for your comment about junior engineers, see kennywinker's reply to this thread - I share the same thoughts.

Very generous of you to blame the screw up of one of the largest companies in the world on a jr engineer.

I’ve been a jr engineer at a large company. I had the power to implement absolutely jack shit on my own. I deeply doubt the security flow for account recovery in meta ai account security was a single jr engineer.

What i think is actually going on is basically a soft form of ai psychosis. Senior engineer gets ai to code ai account recovery feature, that same or a different engineer asks ai to review the feature, and then it gets pushed to prod. Move fast, break things. The ai coded it, the ai reviewed it - the people trusted the ai because it sounds confidently right.

Just like how the ai doesn’t know if you should walk or drive to the car wash, the ai doesn’t understand exploits like this one.

If a single junior engineer can do this, it’s an even bigger indictment of Facebook’s senior management than this exploit. A well-designed system doesn’t rely on individuals never making mistakes and if our hypothetical junior developer can make critical security policy changes without oversight, that should be a C-level job loss event.

If our goal isn’t to make excuses for the top of the org chart, a more likely explanation is that senior management is heavily incentivizing shipping AI features and this went out as a high-impact change reviewed in a rush, probably by AI.

Watch the ageism there, older devs can be lazy and ignorant of security too! (And are responsible for building a dev process that catches such things in review - which points to larger systemic issues over there)

I will agree that anyone that works at Meta is likely not somebody who values privacy very much, though.

...yeah, but its CEO is also who he is. The guy who refers to people using his products as "dumb fucks". That's kind of important

I think he likely means "code that is hand-reviewed" and not directly controlled by the agent. He's probably meaning to differentiate it against the in-process agent writing the code. It doesn't matter too much if that fixed code was written by an LLM under guidance and review of the SWE, outside the agent.

Maybe not hand-written, but definitely static, and at least human-reviewed/tested to only allow sending to previously-validated email addresses.

I can't take Meta seriously, period.