This is treating the LLM like it is the computer or has some kind of way of thinking. But LLM is a "language" model, I'm pretty sure the easier for human to read, the easier for LLM to learn and generate. Abstractions also benefit the model, it does not need to generate a working 2s complement, just a working call to addition of abstracted types.

And just in my experience, I feel everyone is slowly learning, all models are better at the common thing, they are better at bash, they are better at Python and JS, and so on. Everyone trying to invent at that layer has failed to beat that truth. That bootstrapping challenge is dismissed much too easily in the article in my opinion.

▲

kesor 10 hours ago | parent [-]

Binary bits are also a language. A structured language that transistor-based computers execute into some result we humans find valuable. Why wouldn't a model be able to write these binary instructions directly? Why do we need all these layers in between? We don't.

▲

didibus 8 hours ago | parent [-]

Because the learned function to generate binary code is likely more complex than that for Python.

I admit I can't say for sure until we try it. If someone were to train a model at the same scale on the same amount of raw binary code as we do these models on raw language and code, would it perform better at generating working programs. Thing is, it would now fail to understand human language prompts.

From what I know and understand though, it seems like it would be more complex to achieve.

My meta point is, you shouldn't think of it as what would a computer most likely understand, because we're not talking about a CPU/GPU. You have to think, what would a transformer architecture deep neural net better learn and infer? Python or binary code? And I think from that lens it seems more likely it's Python.

	▲	kesor 8 hours ago \| parent [-]
		Why would you think its more complex? There are less permutations of generating transistor on/off states than there are all the different programming languages in use that result in the exact same bits. Who said that creating bits efficiently from English to be computed by CPUs or GPUs must be done with transformer architecture? Maybe it can be, maybe there are other ways of doing it that are better. The AI model architecture is not the focus of the discussion. It is the possibilities of how it can look like if we ask for some computation, and that computation appears without all the middle-men layers we have right now, English->Model->Computation, not English->Model->DSL->Compiler->Linker->Computation.