This doesn't seem like a very good solution to be honest. And why use an LLM for this? What if I want a legit random ass string as my username?

You're not owed anything, you can take your legit random ass string username to another company that allows it.

I suspect any company would take this trade off, losing some customers but significantly lowering fraud.