The other factor that might be in play is that the people who made great games got lucky and didn't really know exactly what made their games so good and aren't able to recapture the magic.

It's a combination of all those things really. There's also a lot of people in commercial game development that get stuck in their thesis of success when the market has moved on.

In particular I'd say right now it's really hard to look at any given game that has commerical potential and spot the winners across any size category. There is a glut of supply that tends to exentuate the hit driven nature of making games as gamers latch onto the next viral success whilst at the same time making it hard to have multiple hits. Back in the day competition was less and money could buy a scope and technology advantage. Both of which are ideas that no longer really apply.