People seem to expect a sudden shift with "self-improvement", but don't AIs already improve themselves via training? What is there to improve?

The human brain operates at better levels of intelligence than the best LLMs, at 20 watts of power. We are a long way to that kind of efficiency, it will probably take both bespoke hardware and algorithmic improvements to catch up to nature.

Only if you think in terms of perceived raw intelligence, but self-update is a form of valuable self-improvement that could benefit current models a lot, if they could commit facts from context into their weights cheaply and reliably.

I think the idea is fundamentally improved architectures. For example, transformer-based models were an incredible stepwise improvement. Self improvement would be a model discovering a stepwise improvement similar to the transformer. And presumably the improved models from that would be more likely to make further advances still.

Learning from training data is technically self-improvement but not the sort that is typically meant in this context.

> AIs already improve themselves via training

Marginally. Model collapse is still a problem. Continuous learning is still a problem.

For AI to make a big leap we need a big break through.