> but without any lived experience perhaps that's an unreasonable expectation.

I don't think that follows. We learn via our perception (i.e. our inputs from the outside world); our perception is mediated by some kind of filter, and memories get written or altered. An LLM on the other hand has static weights. You can provide an LLM a "facts.md" (or whatever) file that "teaches" it stuff that's not in its weights (via the context), and it can use the combination to produce something, but that set of information never modifies its core weights. You could imagine a robot with an LLM inside it that has all kinds of sensors, and processes that write the observations from those sensors into a bunch of databases which are RAG'ed into prompts to simulate learning and experiences, perhaps (jefe likes her coffee with almond milk). But my point is that, architecturally, they don't learn - and that's the bigger obstacle.

PS garcinias seem to be generally delicious.

Neither does my amygdala reshape itself and move closer to my frontal lobe, nor do my cells change their DNAs (minus damage).

Just because "the substrate" has some fixed parts doesn't materially change the equation, I think. Going even more abstract, if I have two Turing complete machines, just because one is Turing complete via some self-modification, or through some other means (e.g. game of life vs Turing machine whose states are fixed) still make them equally powerful.

Don't get me wrong, you are right to point out this shortcoming of LLMs, and I certainly don't believe we are singularity-level. I just don't think this is a good enough argument to prove these statements. A sufficiently smart intelligence that is somehow "fixed", but that can "write on paper" put in a box can't really be distinguished from a "self-modifying" intelligence. In the "worst case" it can just write down a self-modifying intelligence itself as data (since it's sufficiently smart) and execute it to emulate any kind of learning.