No benchmarks, no info on which models are used, ai generated video, just a signup page with nothing else.

Anyhow, this kinda reminds me of that quote about architecture: "We replaced our monolith with micro services so that every outage could be more like a murder mystery."

▲

adam_rida 3 hours ago | parent | next [-]

The evaluator is public here: https://echo.tracerml.ai/eval/

It currently exposes 907 stored rows across seven benchmark families, with prompts, outputs, grades, and cost records. More benchmarks are coming soon.

Echo does not disclose its per-request routing decision because that policy is the product. We can, however, publish some of the eligible open-weight model pool, version dates, aggregate allocation mix, and evaluation settings without exposing the request-level recipe.

New video is also being made.

	▲	seizethecheese 40 minutes ago \| parent [-]
		Isn't the relevant benchmark RouterBench? https://arxiv.org/html/2403.12031v2#S7

▲

guessmyname 4 hours ago | parent | prev | next [-]

> No benchmarks, no info on which models are used, […]

The benchmarks are here → https://echo.tracerml.ai/eval/

They are not good benchmarks but at least they exist.

▲

seizethecheese 3 hours ago | parent [-]

I've been working on a similar project and I found that it's easy to replicate Fable results if you use saturated benchmarks.

In my project, I wasted a huge amount of time trying to improve GPQA Diamond results above ~93% range. I realized my mistake when Fable dropped and made no improvement on this benchmark vs. Opus.

▲

yorwba 30 minutes ago | parent [-]

I wouldn't be surprised if ≈7% of GPQA Diamond questions simply have the wrong answer in the ground truth data, so that getting such a question correct is graded as an error. Most machine-learning benchmarks are rather badly validated.

	▲	seizethecheese 28 minutes ago \| parent [-]
		Yep! I found this interesting article after banging my head against a wall for a long time: https://epoch.ai/gradient-updates/gpqa-diamond-whats-left

▲

codekansas 3 hours ago | parent | prev | next [-]

It's basically trying to replicate OpenRouter, which works pretty well and has a lot of nice features to abstract away any single provider, such as failover, metering, autoswitching, etc. It's actually a really smart infrastructure abstraction.

I just wish this were solving an actual problem rather than being a fairly transparent attempt to say something approximating, "Hey VCs, OpenRouter just became a unicorn but I can basically vibe code it"

Calling it "Fable-level" feels intellectually lazy / dishonest, but then again, what do you expect when there's so much money on the table.

	▲	vunderba an hour ago \| parent \| next [-]
		It feels a bit more like Fugu to me, which acts as a multi-LLM orchestrator (though I think Fugu combines open- and closed-weight models), but without being able to see the “secret sauce” behind how any of them decides the number of "plies" each model in the swarm gets, they all feel rather difficult to compare beyond the big public benchmarks... https://github.com/SakanaAI/fugu
	▲	seizethecheese 2 hours ago \| parent \| prev [-]
		OpenRouter's model router for coding isn't really sophisticated. This is very different.

▲

dimitrios1 2 hours ago | parent | prev | next [-]

Unrelated - but reminds me of my favorite quote by tenderlove:

"microservices turn function calls into distributed computing problems"

	▲	WD-42 2 hours ago \| parent [-]
		grug wonder why big brain take hardest problem, factoring system correctly, and introduce network call too seem very confusing to grug

▲

j45 3 hours ago | parent | prev [-]

It's easy enough to copy and paste in a prompt, no?

Eval tests while giving general indicators might not be similar for each use case.