The Win8 and Metro design disaster is what happens when you give UX free rein, instead of focusing on users they try to start design trends to impress other UX / designers (essential for their career).

I wonder how much of Apples design was basically ‘if you confuse Steve Jobs you’re fired.’ And this acted as a necessary governing force to counteract the need to impress peers.

Metro was a wonderful design for the media player app it was made for. It's great for menu-heavy interactions, not so much for representing stateful things like options and checkboxes and such. Metro isn't the problem, it's trying to shoehorn UIs into it regardless of fit that is.

Part of UX is leveraging what users are already familiar with.

These are all symptoms of a larger problem: very few people care about the users, and you have instead classes of workers living in a bubble, working towards either micro-optimizing metrics or trying to achieve what in their minds is the "ideal" product, pushing the latest fashions of their branch.

So UX engineers will unleash the latest fad (see Apple's glass UI, or Material Design, variations of flat gray design, etc...), PMs will insist in dumbing down UIs, engineers will push whatever micro-service architecture because it's "cool" or push for rewrites in Rust / Typescript. At the same time, it's very rare for companies to have a single person (or restricted group of people) with a global view on what the product line is trying to achieve long-term.