You're giving a lot of credit to the human alternative, especially considering that the attacker only needs to find one lazy human.

Still makes this exponentially worse, no? It works every time and it's automated so scales up as quickly as you're able to request it.

Come on, this attack vector would have been flagged by at least one person and you won’t then have multiple accounts hacked because of it. AI reacts fairly predictably to a single attack vector and don’t learn unless it gets flagged and then taught.