44% on ARC-AGI-1 in 67 cents

pwmglenn a minute ago | parent | next [-]

Really impressive and creative research. I wonder if the leading labs do anything similar with their models? It doenst look like the open source labs do?

▲

larodi 5 minutes ago | parent | prev | next [-]

"I don’t understand why others didn’t figure this out"

- how about we allot the possibility that so many of presumed ML experts don't have any clue what they be doing, and are eventually API bitches, nothing more.

▲

xeonax 43 minutes ago | parent | prev | next [-]

Even cooler is his about me mention of saving his own life https://mvakde.github.io/ > Saved myself in a medical emergency (doctors didn't know what rhabdomyolysis was)

▲

qlm 9 minutes ago | parent [-]

Crazy, considering rhabdo isn't that rare.

	▲	p-e-w a minute ago \| parent [-]
		“What do you call a medical student who graduated at the bottom of their class?” “Doctor.”

▲

embedding-shape an hour ago | parent | prev | next [-]

Is the author only running their model against one benchmark? I don't think anyone finds that difficult to achieve, the difficulty comes when you want to make the model not benchmaxxed to a specific benchmark, and generalize so it can solve problems not part of the training data, but seems this model is specifically for not this? How useful is that?

If you just wanted to pass these specific tasks in this specific benchmark, and wanted to do so cheaply, I'm sure a non-LLM-based approach would yield better results for even cheaper, since what the author's model does, seem to basically be "solve ARC puzzles", not a general LLM or "coding" LLM.

▲

bkaae 22 minutes ago | parent | next [-]

I read this as a response to the current hype around LLMs. He is showing computers can solve these issues, without using an LLM architecture. A lot of people have sort of forgot that machine learning is more than just LLMs these days.

I found it to be a very interesting angle.

	▲	embedding-shape 17 minutes ago \| parent [-]
		> He is showing computers can solve these issues, without using an LLM architecture. Isn't it a LLM he's building though? My very point is that this particular use case could be solved better without building a LLM, now you claim he is not? The description of what he's doing surely makes it sound like it's a (very small) LLM, and personally I'm still on the "if it quacks like a duck" train in life. > A lot of people have sort of forgot that machine learning is more than just LLMs these days. Yeah, which I guess if you make my previous comment more concise, is exactly what I state too.

▲

f311a 25 minutes ago | parent | prev [-]

The whole point of his model is to optimize for a very specific benchmark.

BUT, he does not use labels when training, so the model does not know the answers.

▲

embedding-shape 17 minutes ago | parent [-]

> The whole point of his model is to optimize for a very specific benchmark.

But benchmaxxing is what we generally try to avoid for training, as there is no point really for it. We used to call it "overfitting", now you're saying this person does it intentionally? Why?

	▲	f311a 10 minutes ago \| parent [-]
		Why not? There is $700k reward for the next iteration of this benchmark https://www.kaggle.com/competitions/arc-prize-2026-arc-agi-2... I would not call this overfitting, it's finetuning for specific task where you have a benchmark.

▲

eis 33 minutes ago | parent | prev [-]

> Increases in LLM scores are now mainly driven by post training (evidence in next section) and are probably a function of amount of synthetic data. They are learning to solve ARC tasks, not learn general abstract reasoning

Agreed and that's for any benchmark. Private tests are better but you still have to trust the provider to not log and use them for training.

That's why I like when a new set of tests like a new ARC-AGI version is published, that's where you can see which of the models abstracted to more general capabilities instead of being focused on the previous tasks. Most models completely fail new ARC-AGI tests.

The "67 cents" part though is misleading imho. You can't extrapolate from there and think that investing say $100 will get you a lot better results. You hit a ceiling very fast and investing into more compute will give you diminishing results. So yes, you can train a custom model to do somewhat decently on a specific set of tasks but then what?

	▲	bkaae 20 minutes ago \| parent [-]
		Then nothing - that's awesome. People think that LLMs are the know-all do-all solution to every problem now. Putting solutions in terms of cents is a great way to potentially win over some ai boosters imo. There are other ways to solve hard problems.