It can learn very well within its context. And it can leave notes for itself to read into future contexts.

That's learning, whether you agree or not, and whether you like it or not. Baby steps, yes, but walking nevertheless.

There seems to be very few cases where ICL is really doing something that can be considered as learning (albeit ephemeral) rather than just utilizing in-context data via induction heads.

Yes, I'm aware of the linear regression example, and a few others, but these appear to just be specific capabilities that were learnt during pre-training, presumably pursuant to reducing errors on similar-but-different training samples, not any kind of generic run-time learning capability.