It wasn't even ten years ago when I was participating in user studies as a developer at MS. And it was the real deal: we had a bunch of people, specifically selected to give a diverse cross-cut of the user base (so varying backgrounds, experience level etc), sit down and try to do some simple tasks with our product, while devs and PMs watched over a camera. And, crucially, you couldn't ask questions or provide guidance during that time - only after they were done. That was incredibly informative, much more so than any telemetry I've seen before or after, and I wish that was the norm in all companies; but, in any case, Microsoft definitely had both the resources and the inclination to do that.

The problem is that no amount of studies or A/B testing is going to change a political decision inside the company. And with AI, I'm convinced that for all the big players it is political at this point simply because all the execs have bet so much money on it. If they can't make it work, we're talking about literally billions of dollars of responsibility. Hence these desperate attempts to shove it everywhere in hopes that something somewhere would work well enough, if not to recoup the investment, then at least to postpone the moment it all comes crashing down.