Remix.run Logo
MallocVoidstar 2 hours ago

> Claude Code 2.1.207 and OpenCode 1.17.18, both pinned to claude-sonnet-4-5

So not only is this article AI-written, but the testing was entirely done by AI, too? I can't see any other reason to use such an old model.

> Our traffic passes through a local LLM gateway that wraps requests in its own envelope, a constant we measured at roughly 6,200 tokens with bare calibration requests

Why do you need to do calibration requests to figure out how your own gateway is affecting requests?

> Its subagent lane did not complete cleanly through our gateway

> We attempted to toggle extended thinking in both harnesses and are declining to publish numbers. Our gateway applies its own thinking policy, neither harness's toggle demonstrably survived the path, and anything we quoted would be noise.

Why is your own gateway screwing with your testing?

systima 2 hours ago | parent [-]

Model:

Cost, mainly. The runs went through a Claude Max subscription rather than metered API billing, and pinning an older stable snapshot kept run-to-run comparisons clean and cheap. The fixed harness payload (system prompt plus tool schemas), so the headline numbers shouldn't change too much.

That said, happy to re-run the matrix on Fable and publish the diff; payload figures should barely move, tool-calling behaviour might.

Gateway:

Meridian (github.com/rynfar/meridian); proxy that bridges the Claude Code SDK to a standard Anthropic endpoint so a Claude Max subscription can drive OpenCode-et-al.

It's the auth route for all agent traffic on the machine, not something built for the benchmark.