I worked in NY publishing in the 1990s and also did some small press stuff, and even then the 'death of the midlist' was already an old topic. And yeah, consolidation had someting to do with it: publishers were owned by bigger businesses that saw them as black boxes to extract value from. Distribution was changing: the big 'superstores' and Amazon/online sales starting to be a thing. Mass-market was getting crushed. Obviously, not everything that got published was a bestseller, or even expected to be, but authors couldn't get the same space to grow a career. If it didn't work, they'd be cut.

Now I'm a production editor for a uni press. For a while, it seemed to be a bit of a haven from the madness, but it's coming for us now too.