Remix.run Logo
▲ ricardobeat an hour ago

Neon / Carbon / Argon / Hydrogen are all in the same ballpark, why did they even release four different models?

Neon doesn't seem particularly strong at coding vs Argon, and Opus 5.5 still beats both in my experience. The neurons it lost that made it incompetent at English seem to have been fully dedicated to being a freak at engineering.

▲brainwad 10 minutes ago | parent | next [-]

Argon is fully obsoleted by Carbon. The other two are smaller and thus faster/cheaper.

▲derefr an hour ago | parent | prev | next [-]

In general, one could assert without knowing any details that:

1. the models likely have orders-of-magnitude different backend costs to run (e.g. each warmed instance of the model reserving one slice of one GPU, vs one full GPU, vs layers distributed across multiple GPUs) -- as otherwise it wouldn't be worth explicitly distinguishing these models and putting different token spends on them (they'd instead just be automatically routed between); and

2. there's at least one particular specialized problem domain, not captured by this benchmark yet still relevant to external customers (so not just "it's better at training data evaluation/synthesis"), where these models are more-competent-enough to make their higher per-token costs "worth it" to those external customers.

I won't hazard a guess at what that problem domain is; I'm sure it'll be deduced after people spend a few weeks playing with it.

▲anon7000 25 minutes ago | parent | prev | next [-]

I mean, I was very impressed with Neon over a couple days vs Opus 5.5. It accomplished tasks faster, found root causes of problems faster, wayyy nicer to talk to, didn’t create super flowery PR descriptions. I would say it had moderately difficult tasks. Not super simple, but also not wildly difficult or broad in scope.

It doesn’t feel actually dumb, like many other models. (Definitely smarter than Sonnet which was already ok.) And it’s cheaper. So I think it’ll be my go to for most tasks

▲raincole an hour ago | parent | prev [-]

Because benchmarks aren't everything? If you see Neon and Hydrogen get the same score in a benchmark you should question if the benchmark measures what we care instead of suspecting that Neon and Hydrogen are equally powerful.

▲ricardobeat an hour ago | parent [-]

This is based on my (admittedly short) experience using them in agy, not the benchmarks alone.

▲CAP_NET_ADMIN an hour ago | parent [-]

Maybe there's something wrong with your harness/docs - I've been able to achieve better results with Neon high than Opus 5.5 high.

▲ricardobeat an hour ago | parent [-]

I'm currently doing a round of fairly simple changes - refactoring some code based on different memory allocation strategies. Neon (high) is working on a single file < 1000 lines, Opus (low) reviews, I'm manually coordinating, using their own harnesses, agy/CC.

n = ±10, but Opus always catches a subtle bug, a leftover comment, a missed edge case, naming inconsistency, tooling gap. Which Neon seems oblivious to. So far 100% of its reviews have been correct; accepted, fixed and validated by Neon.

Performance is highly dependent on the task/domain, yes, but you'd expect at least one of those four models, the more expensive one, to catch up to Opus in the coding department, but that does not seem to be the case.

▲disgruntledphd2 40 minutes ago | parent [-]

What happens if you swap the models writing and reviewing? So Opus implementor, Neon reviewer.

▲ricardobeat a minute ago | parent [-]

Ok, I had some time to kill :)

The issue: "temporary buffers (out_buf, cps, out_cps, src_cps) are manually freed across multiple return branches. Pairing each acquisition with defer mem::free(...) — or using @pool() + mem::temp_array for the short-lived out_buf / out_cps scratch arrays — removes the branch-by-branch cleanup".

## Round 1

Neon implements. ~6 minutes.

Opus review. 30s. Prompt is simply "review the unstaged changes".

- The heap_buf + defer if (heap_buf != null) combination is clunky, much like the committed flag was. C3 has a built-in way to do this

- if no character actually changes, return the string itself to skip the buffer

- Allocator mismatch: unicode_normalize_simple in src/unicode_wrapper.c allocates with libc realloc, but result_cps is released with mem::free

## Round 2

Opus implements. 2m44s.

Neon reviews: 4m30s.

"Looks good — no bugs or leaks introduced."