Remix.run Logo
▲ raincole an hour ago

Because benchmarks aren't everything? If you see Neon and Hydrogen get the same score in a benchmark you should question if the benchmark measures what we care instead of suspecting that Neon and Hydrogen are equally powerful.

▲ricardobeat an hour ago | parent [-]

This is based on my (admittedly short) experience using them in agy, not the benchmarks alone.

▲CAP_NET_ADMIN an hour ago | parent [-]

Maybe there's something wrong with your harness/docs - I've been able to achieve better results with Neon high than Opus 5.5 high.

▲ricardobeat an hour ago | parent [-]

I'm currently doing a round of fairly simple changes - refactoring some code based on different memory allocation strategies. Neon (high) is working on a single file < 1000 lines, Opus (low) reviews, I'm manually coordinating, using their own harnesses, agy/CC.

n = ±10, but Opus always catches a subtle bug, a leftover comment, a missed edge case, naming inconsistency, tooling gap. Which Neon seems oblivious to. So far 100% of its reviews have been correct; accepted, fixed and validated by Neon.

Performance is highly dependent on the task/domain, yes, but you'd expect at least one of those four models, the more expensive one, to catch up to Opus in the coding department, but that does not seem to be the case.

▲disgruntledphd2 39 minutes ago | parent [-]

What happens if you swap the models writing and reviewing? So Opus implementor, Neon reviewer.