The AI part does seem relevant because it enabled incredibly low-effort “social” engineering.

For what it’s worth I don’t think you can call this social engineering since there was no human on the other end, even though it appears similar.

The question is, if there were actual human support agents, would they have built additional safeguards to prevent social engineering in this manner?

One concerning feature of AI is the speed and volume it is capable of failing at if poorly controlled, whether or not it’s more accurate than humans.

Even if humans failed at the same rate, if you tried to exploit at scale you’d be throttled by the size of the support team. The failure would happen at human-scale time frames and throughput.

a human would have noticed something different about the requests it was getting, or the frequency of requests, and as soon as it noticed a shift, it would have carried that knowledge forward and intensified the scrutiny if something seemed off- eventually communicating it up the chain.

- instead of the ai context dying.

in the ai case, information only survives to the extent where the ai is empowered to store a note or notify a manager of an observation. Anything that does not result in sending a message/storage is wiped

Why did the account recovery system need AI. Surely just an email would do? What added value would AI add?

There's no social engineering here, since all they have to do is copy and paste. This is a complete process design fail.