Neither does my amygdala reshape itself and move closer to my frontal lobe, nor do my cells change their DNAs (minus damage).

Just because "the substrate" has some fixed parts doesn't materially change the equation, I think. Going even more abstract, if I have two Turing complete machines, just because one is Turing complete via some self-modification, or through some other means (e.g. game of life vs Turing machine whose states are fixed) still make them equally powerful.

Don't get me wrong, you are right to point out this shortcoming of LLMs, and I certainly don't believe we are singularity-level. I just don't think this is a good enough argument to prove these statements. A sufficiently smart intelligence that is somehow "fixed", but that can "write on paper" put in a box can't really be distinguished from a "self-modifying" intelligence. In the "worst case" it can just write down a self-modifying intelligence itself as data (since it's sufficiently smart) and execute it to emulate any kind of learning.