I read this as a response to the current hype around LLMs. He is showing computers can solve these issues, without using an LLM architecture. A lot of people have sort of forgot that machine learning is more than just LLMs these days.

I found it to be a very interesting angle.

▲

embedding-shape a day ago | parent | next [-]

> He is showing computers can solve these issues, without using an LLM architecture.

Isn't it a LLM he's building though? My very point is that this particular use case could be solved better without building a LLM, now you claim he is not? The description of what he's doing surely makes it sound like it's a (very small) LLM, and personally I'm still on the "if it quacks like a duck" train in life.

> A lot of people have sort of forgot that machine learning is more than just LLMs these days.

Yeah, which I guess if you make my previous comment more concise, is exactly what I state too.

▲

dakolli a day ago | parent [-]

Nowhere does he say he built an llm. Hes using a transformer, not an llm.

▲

embedding-shape a day ago | parent [-]

> Nowhere does he say he built an llm. Hes using a transformer, not an llm.

Please describe what in your mind a "LLM" is exactly, then describe what this person is building. To me this sounds like "He's not building a calculator, he's just building a program that can do addition, minus, multiplication and division and display the results".

Obviously it's not a Large Language Model, but to me this looks more like a LLM than not, given the architecture he's chosen. But again, maybe I misunderstand?

▲

evilmathkid a day ago | parent | next [-]

Its not an LLM if there's no pretraining. AR transformers were around before LLMs and will be there after LLMs.

When I made this, the point was to show that you dont need pretraining (which is what makes an LLM) to perform well on complex tasks

And yes it is not a language model either. I did not train it on any language data. Only ARC puzzles

▲

soVeryTired a day ago | parent [-]

Out of interest, would you call BERT an LLM? It’s pre trained but not particularly large.

	▲	stymaar a day ago \| parent \| next [-]
		AFAIK, the “large” qualifier came when transformers allowed to scale the size of language models compared to the recurrent models that where in fashion before. And although BERT isn't large by today's standard, it was large enough for the time.
	▲	evilmathkid a day ago \| parent \| prev [-]
		idk the definition is fuzzy. thats why people use the "modern" qualifier to talk about decoder-only style and this is also not clean since you now have reasoning models which are separate

▲

soVeryTired a day ago | parent | prev | next [-]

It’s neither large nor language-based. ARC-AGI-1 is grid-based and nonverbal.

Use of a transformer is not necessary or sufficient to qualify as an LLM.

▲

crotonix a day ago | parent | prev | next [-]

Transformer solves a Seq2Seq problem just like RNNs. All Seq2Seq problems need not involve a language. In this case teaching on ARC puzzles doesn't mean what he trained is now trained on a language which will be English(or any other language) in this case. So, does his training successfully models "English as a language" -> No. This implies it is not "Large" and has not modeled any "language".

▲

yorwba a day ago | parent | prev [-]

A LLM should at the very least be a language model, i.e. be able to take human-readable text as input or produce it as output. Transformers are used for plenty of tasks that don't involve language, for example object detection or blind source separation, where the models aren't called LMs; and on the other hand there are some LLM architectures that exclusively use linear attention variants and aren't really transformers anymore.

▲

bonplan23 a day ago | parent | prev [-]

Just to be clear: It was well known that you can reach such scores with small models and without an LLM if you train on the task. The author highlights those models himself - e.g. HRM/TRM.

The novelty is more that it works with such a plain transformer and low compute price.