I'm not saying they are alive, they have consciousness, etc.

But do we really know how LLM's(and the layers above, harness, RL, etc) work, or is there some emergent, surprising phenomena in those systems?

There was an emergent and surprising phenomena in collaboratively writing Linux kernel and low level backbone of modern operating systems, or developing mobile apps, or scaling Internet to broader population.

What is different about LLMs? Is it their social aspect and influence? All three examples above were also social to a different extent.