It still puzzles me decades later how MS built the most functional, intuitive and optimised desktop environment possible then simply threw it away

It still is if you're an enterprise customer. The retail users aren't Microsoft's cash cows, so they get ads and BS in their editions. The underlying APIs are still stable and MS provides the LTSC & Server editions to businesses which lack all that retail cruft.

Idk why they use Electron for everything, they literally built the UI stack itself and C# is insanely good at building UIs if they stop trying to reinvent UIs in C# that is.

The pivot point was Windows 95.

Competition. In the first half of the 90s Windows faced a lot more of it. Then they didn't, and standards slipped. Why invest in Windows when people will buy it anyway?

Upgrades. In the first half of the 90s Windows was mostly software bought by PC users directly, rather than getting it with the hardware. So, if you could make Windows 95 run in 4mb of RAM rather than 8mb of RAM, you'd make way more sales on release day. As the industry matured, this model disappeared in favor of one where users got the OS with their hardware purchase and rarely bought upgrades, then never bought them, then never even upgraded when offered them for free. This inverted the incentive to optimize because now the customer was the OEMs, not the end user. Not optimizing as aggressively naturally came out of that because the only new sales of Windows would be on new machines with the newest specs, and OEMs wanted MS to give users reasons to buy new hardware anyway.

UI testing. In the 1990s the desktop GUI paradigm was new and Apple's competitive advantage was UI quality, so Microsoft ran lots of usability studies to figure out what worked. It wasn't a cultural problem because most UI was designed by programmers who freely admitted they didn't really know what worked. The reason the start button had "Start" written on it was because of these tests. After Windows 95 the culture of usability studies disappeared, as they might imply that the professional designers didn't know what they were doing, and those designers came to compete on looks. Also it just got a lot harder to change the basic desktop UI designs anyway.

The web. When people mostly wrote Windows apps, investing in Windows itself made sense. Once everyone migrated to web apps it made much less sense. Data is no longer stored in files locally so making Explorer more powerful doesn't help, it makes more sense to simplify it. There's no longer any concept of a Windows app so adding new APIs is low ROI outside of gaming, as the only consumer is the browser. As a consequence all the people with ambition abandoned the Windows team to work on web-related stuff like Azure, where you could have actual impact. The 90s Windows/MacOS teams were full of people thinking big thoughts about how to write better software hence stuff like DCOM, OpenDoc, QuickTime, DirectMusic and so on. The overwhelming preference of developers for making websites regardless of the preferences of the users meant developing new OS ideas was a waste of time; browsers would not expose these features, so devs wouldn't use them, so apps wouldn't require them, so users would buy new computers to get access to them.

And that's why MS threw Windows away. It simply isn't a valuable asset anymore.

It's quite common for a company to build a good product and then once the initial wave of ICs and management moves on, the next waves of employees either don't understand what they're maintaining or simply don't care because they see a chance to extract short term gains from the built-up intellectual capital others generated.

It's functional - yes, intuitive - maybe, but optimized is highly debatable.

The answer to maintaining a highly functional and stable OS is piles and piles of backwards compatibility misery on the devs.

You want Windows 9? Sorry, some code checks the string for Windows 9 to determine if the OS is Windows 95 or 98.

Piracy. The consumer versions are filled with ads because most people don't pay for them.