The off-the-shelf player in this space (structured red-team search) is probably https://github.com/usestrix/strix (no affiliation). But any frontier model, if you tell it "here's how the authentication system works, go looking for bugs" will probably do a good job.

Curious how Strix/Shannon/Deepteam all stack up against one another