No, we did a actual test using our existing testing framework. We have shitloads of metrics to know when a user gets stuck, when they give up, which login path they took, etc.

This wasn't a half assed test but a legitimate effort to improve something that we never prioritized

We had a legitimate 25% reduction in users giving up logging in in a system that has millions of users.

We ran a 50-50 AB test for several weeks to confirm the data and then turned it on completely

edit: If you haven't already read my post, I'd also like to say that the benefit AI gives us is that I worked on something I never get to work on, the analyst got to try a hunch he always had, and we got to see it go live in a day. If it didn't' work out, we were out a day of work which beats the few weeks of an effort prior to AI that we would spend on something just to find out it didn't work.