Yeah, there's some beautiful math underlying what LLMs are doing, and it's the same math our neocortex runs on.

burnte 2 minutes ago | parent | next [-]

> and it's the same math our neocortex runs on.

The math is based upon theories of how the brain work, but even if those theories are right, this math is a great simplification and subset of what organic brains do.

▲

dekhn an hour ago | parent | prev | next [-]

There's plenty of beautiful math there, but the relationship to what our neocortex does is pretty distant. Individual biological neurons can do fairly complicated things, including compute 10-bit parity functions (you would normally need a 3-layer MLP with a bunch of digital neurons to do this). And they don't seem to use backpropagation for learning.

▲

ainch 2 hours ago | parent | prev [-]

Could you provide more detail? My understanding is that the neocortex is predominantly focused on forwards simulation, which seems distinct to how transformers operate.

	▲	koverstreet an hour ago \| parent [-]
		That's fundamental to how anything that compresses/understands the world has to work, in the Kolmogeravian sense. That's why people denigrate LLMs as being just "next token predictors" - they're not wrong, but they're missing the point. Because to do that kind of prediction out in the world you have to build up an accurate model of reality - a model that includes yourself! Which is why we and LLMs are self aware. For the "how", it's been known for some time that LLMs operate on a Reimannian manifold - the semantic manifold - and that's a good place to start if you want to learn how they actually work; how a Reimannian manifold (plus some extra structure on top) can represent natural language in a form you can do work with is the part I find particularly beautiful. At a high level, the neocortex and LLMs appear to compute on the manifold in basically the same way - though a lot of the details are different; both are more sophisticated in some areas and less in others.