LLMs really can't be improved all that much beyond what we currently have, because they're fundamentally limited by their architecture, which is what ultimately leads to this sort of behaviour.

Unfortunately the AI bubble seems to be predicated on just improving LLMs and really really hoping that they'll magically turn into even weakly general AIs (or even AGIs like the worst Kool-aid drinkers claim they will), so everybody is throwing absolutely bonkers amounts of money at incremental improvements to existing architectures, instead of doing the hard thing and trying to come up with better architectures.

I doubt static networks like LLMs (or practically all other neural networks that are currently in use) will ever be candidates for general AI. All they can do is react to external input, they don't have any sort of an "inner life" outside of that, ie. the network isn't active except when you throw input at it. They literally can't even learn, and (re)training them takes ridiculous amounts of money and compute.

I'd wager that for producing an actual AGI, spiking neural networks or something similar to them would be what you'd want to lean in to, maybe with some kind of neuroplasticity-like mechanism. Spiking networks already exist and they can do some pretty cool stuff, but nowhere near what LLMs can do right now (even if they do do it kinda badly). Currently they're harder to train than more traditional static NNs because they're not differentiable so you can't do backpropagation, and they're still relatively new so there's a lot of open questions about eg. the uses and benefits of different neural models and such.

I think there is something to be said about the value of bad information. For example, pre ai, how might you come to the correct answer for something? You might dig into the underlying documentation or whatever "primary literature" exist for that thing and get the correct answer.

However, that was never very many people. Only the smart ones. Many would prefer to have shouted into the void at reddit/stackoverflow/quora/yahoo answers/forums/irc/whatever, to seek an "easy" answer that is probably not entirely correct if you bothered going right to the source of truth.

That represents a ton of money controlling that pipeline and selling expensive monthly subscriptions to people to use it. Even better if you can shoehorn yourself into the workplace, and get work to pay for it at a premium per user. Get people to come to rely on it and have no clue how to deal with anything without it.

It doesn't matter if it's any good. That isn't even the point. It just has to be the first thing people reach for and therefore available to every consumer and worker, a mandatory subscription most people now feel obliged to pay for.

This is why these companies are worth billions. Not for the utility, but from the money to be made off of the people who don't know any better.