AI2: Open Coding Agents

ahmadyan an hour ago | parent | next [-]

Claims in the article are incorrect. They conveniently ignore Meta CWM models, which are open-sourced [1] and open-weight [2] and are at 65% SWE-bench verified (with TTS) and 54% pass@1 and the same size (32B dense). So claims like "surpassing prior open-source state-of-the-art coding models of comparable sizes and context lengths" and conveniently leaving out the previous OSS SOTA out of your eval tables are ... sketch.

[1]https://github.com/facebookresearch/cwm [2]https://huggingface.co/facebook/cwm

▲

ethan_l_shen 42 minutes ago | parent | next [-]

Hey! These are great observations. So first, while TTS can improve performance, we wanted to evaluate the raw capability of our model. This meant generating only one rollout per evaluation instance, which follows other papers in the space like SWE-smith and BugPilot. In addition, TTS adds extra inference cost and is reliant on how rollouts are ranked, two confounding factors for deployable models where memory and inference speed are extremely important.

Following that line of reasoning, context length is another very large confounding factor. Longer context lengths improve performance - but also result in enormous increases in KV cache size and memory requirements. We decide to control for this in our paper and focus at the 32K context length for 32B size models, a context length that already pushes the bounds of what can be "deployable" locally.

Still, we evaluate at 64K context length using YARN and are able to outperform CWM's 54% performance (non TTS), which it achieves using 128K context, a substantial increase over what we use. This is also pretty significant because we only ever train at 32K context, but CWM trains for a full 128K.

▲

philipkglass an hour ago | parent | prev | next [-]

The difference is that the Allen Institute models have open training data, not just open code and weights. Meta doesn't share the training data you would need to reproduce their final models. For many uses open-weight models are nearly as good, but for advancing research it's much better to have everything in the open.

	▲	kevmo314 27 minutes ago \| parent [-]
		Reading their paper, it wasn't trained from scratch, it's a fine tune of a Qwen3-32B model. I think this approach is correct, but it does mean that only a subset of the training data is really open.

▲

mhitza an hour ago | parent | prev [-]

The linked open weight disallows commercial, and is only licensed for research purpose

▲

augusteo 41 minutes ago | parent | prev | next [-]

The ahmadyan comparison is fair. Meta's CWM models hitting 65% vs SERA's 54% is a meaningful gap.

But the interesting number here isn't accuracy. It's the $400 to reproduce top open-source performance. That's the part that matters for teams building internal tooling.

We've been running agents on proprietary codebases at work. The pain isn't model quality. It's customization. Most off-the-shelf agents don't understand your repo structure, your conventions, your test patterns. If you can fine-tune a 32B model on your own codebase for a few hundred dollars, that changes the economics completely.

But codebases changes everyday, so finetuning will have to be continuously done!

Probably not worth it versus something like Claude Code.

Curious whether anyone's tried this on non-Python codebases. Most SWE-Bench stuff is Python-heavy.

	▲	storystarling 35 minutes ago \| parent [-]
		The fine-tuning overhead is definitely a factor, but for smaller shops the hard constraint is usually inference VRAM. Running a 32B model locally or on a rented GPU is surprisingly expensive if you aren't saturating it. Even at 4-bit quantization you are looking at dual 3090s or an A6000 to get decent tokens per second. The $400 training cost is impressive but the hosting bill is what actually kills the margin compared to per-token APIs.

▲

nickandbro 38 minutes ago | parent | prev | next [-]

Great work! Really respect AI2. they open source everything. The model, the weights, the training pipeline, inference stack, and corpus

▲

Imustaskforhelp 27 minutes ago | parent | prev | next [-]

Hey this looks great? Is it available on Openrouter.

I wish if AI2 could release a more denser model on Openrouter for free than the 8B model as I was using Devstral model for agentic purposes.

If we can get an agentic good 32B like model on openrouter for ~free, then I feel like it will be very interesting to see how things would go imo.

Good luck with AI2! The premise of truly open source models is really interesting and I feel like it could help bring more innovation in the space imo!

▲

jauntywundrkind an hour ago | parent | prev | next [-]

Awesome stuff. Output speed looks crazy fast too.

I wonder if this indeed will start prompting more language specific work.

Afaik training still requires not just looking at sample code but also being able to write loss functions being able to have problems the AI can work at. That seems hard.

One random thought, are there training styles of just deleting some code from "good" projects then making the AI make it work again?

	▲	CuriouslyC an hour ago \| parent [-]
		The technique people use is to capture PR diffs from public repos and extract the tests then use that to see if agents can reconstruct the patch that satisfies the tests.

▲

khimaros an hour ago | parent | prev [-]

it's great to see this kind of progress in reproducible weights, but color me confused. this claims to be better and smaller than Devstral-Small-2-24B, while clocking in at 32B (larger) and scoring more poorly?

	▲	ethan_l_shen 39 minutes ago \| parent [-]
		Hey! We are able to outperform Devstral-Small-2-24B when specializing on repositories, and come well within the range of uncertainty with our best SERA-32B model. That being said, our model is a bit larger than Devstral 24B. Could you point out what in the paper gave the impression that we were smaller? If theres something unclear we would love to revise