This assumes that everyone doesn't reach "good enough" at some point. The best developed area here -- software -- seems to suggest a "good enough" plateau, if not an absolute plateau.

I don't think so - they could get a lot better, but it's going to take a lot of human brainpower to get there. They are barely adequate now, but their weaknesses are sold as intrinsic to the nature of the software in a way that mere civilians are too dumb to understand.

But imagine a world in which you could ask AI to write software to accomplish a particular task, and it understood completely what you meant, made tasteful choices, interviewed you as it went along to clarify things, and generated beautifully commented (maybe even Knuth-style literate) code that wasn't dogshit. Doesn't seem like an impossibility. What is annoying is that we're pretending like we have anything like that now.

Instead, there's this sort of science fiction pitch where we pretend like the LLM is just too damn logical to understand your sloppy human language. It's not logical, it's built on (human-generated) slop and generates slop. It's simply failing. We've come up with better ways to orient it towards big picture thinking and filling in outlines, but we don't know how to train it on big pictures.

Not that this would stop open models/harnesses from following (or even leading) frontier labs and destroying their insane valuations, but I don't think there's any evidence that a plateau has been reached. What we have is not great, and while a wall may have been hit on current approaches, the money, compute, and simply current progress in ideas has us in an environment where everything could change over the course of a crazy day. A single blog entry.

I do think that the frontier labs seem to have reached at least a temporary plateau. But if they figured out how to move past it and this new method didn't require a lot more compute, they'd still be screwed. Their only out is to pass the bag to the public, but Firmus, CharacterAI, CoreWeave...