> I gave it the hardest real task that fits on one machine: reverse-engineering a commercial app's license check...

Respectfully, tasks that allow for explicit straightforward true/false or done/not-done tests are not the "hardest real task[s]." In fact, those are the ones that see the most gains from AI-assisted coding.

Testable tasks are where the largest opportunity is.

Maybe so, but there were other elements that I've seen frontier models struggle with in the past, which was the perspective I had coming into this. It's the type of test I run frequently and this is the first small local model I've seen pull it off.

It had a very non-standard RSA key implementation that was obfuscated heavily. As well, it has an online license check at first run, and that part typically trips up most of the local models I've tried. I've been running this test for about a year now with different models, and it was the first I've seen not only figure out the RSA key implementation, but the first that didn't just give up once it saw the online license check. Even though it's only a first-time launch check.

That's why I call it one of the hardest, because in my experience, it has been. It's the first local model I've seen pull it off end-to-end. For some of the reverse engineering work that I've done with LLMs, none have been as consistent as this particular test at highlighting a model's failure in this domain.

I have access to Daybreak Blue and I'm approved for Anthropic's Cybersecurity program, so I might run the same test with both of those just to see, because it's been a while since I used a frontier model on this test. I imagine they'll make relatively light work of it, though, assuming it doesn't trip the relaxed guardrails.

Which is exactly why we saw 1000s of ' "I" rewrote <mature software> in rust' posts last year when agentic coding really took off.

Agents (even ones powered by small models) do reasonably well when provided an oracle to work against.

Also a reverse engineering tasks that can be don with just static analysis is arguably not the hardest reverse engineering task. For those small models I would say it's not about the capabilities but more about the context size it can actually use.

I guess the Riemann Hypothesis is an easy task then.

as he rightly says, fuck all benchmarks and metric as long as it can get <my task> done. who cares what it specifically good at or not and trying to create benchmarks as long as it solves <my problem>.

hardest for claude and chatgpt, which will refuse to do this task haha