As an aside, I do not like unnecessarily misleading graphs. Own your strengths and weaknesses.

In the "Results" section, they choose to compare Ultramarine only to models that it's better than on every metric. They ignore models like Fable 5.1, GPT-6 Astra, Claude Opus 5.5, etc.

Then, in the "Score against cost per task" graph, which is focused on price, suddenly all of those models that perform better than Ultramarine come roaring back because they are also more expensive.

Just list yourselves as the second best sometimes. Your thing is impressive. Don't let the idiot marketing guy hide that behind stupid tricks.

▲

N_Lens 4 minutes ago | parent | next [-]

That looks like extreme cherrypicking and doing this speaks against the Ultramarine team. Virtue is at a premium in these times.

Not going to be trying it just based on this observed behaviour.

▲

lwhi an hour ago | parent | prev | next [-]

Yes, I'm far more likely to purchase if I can understand the full picture. It's not quite as good as frontier models, but it's 60 times cheaper.

Honesty and transparency are attractive.

▲

thatsabadlook an hour ago | parent | next [-]

Maybe you are more likely to buy things when benchmarks are honest, but openai and anthropic have been cheating at those for years. They get purchased a lot. Most purchases are emotional things, even for technical products. That's why they sell llms like it's crack, a weapon, or if you don't you will starve to death in the future. One thing I like about the Chinese and euro labs is they don't market their products in harmful ways, but that's a tangent.

▲

cadamsdotcom 36 minutes ago | parent [-]

Sounds like there's a gap in the market for an honest player to fill.

▲

thatsabadlook 25 minutes ago | parent [-]

It's more or less all stupid anyways. Benchmarks can give someone and understanding of what a model was trained on but they don't explain tone, consistency, the type of answers, the restrictions, what the money is going to, etc.

And ultimately most use of llms is gambling. So a lot of people get hooked on their one machine or whatever despite all reasons to the contrary. Some of the most deranged LLM users I e encountered are hooked on a variety of models but still use ones from like 2 years ago for specific tasks for emotional reasons.

The truth is, for 99.999% of use cases the top 10 models are more or less interchangeable. Either they can mostly do what someone wants, or none of them should probably be used for that application even if s leading lab is advertising it.

	▲	cindyllm 21 minutes ago \| parent [-]
		[dead]

▲

adrithmetiqa 24 minutes ago | parent | prev [-]

I really would like to understand how it can be 60 times cheaper. Is that a like for like comparison? Is there a more equivalent frontier model that is closer in price?

▲

digitcatphd 3 hours ago | parent | prev | next [-]

Yes, for me when they do this I immediately dismiss the model, don't even bother testing it. If you need to lie with statistics your model isn't good enough.

▲

arendtio 4 hours ago | parent | prev | next [-]

I think they just decided to use the formatting different than the others.

Normally, the background highlight color indicates the leader for every benchmark, but they chose to highlight their column even though there are better models.

APEX Agents => Claude Fable 5 and Gemini.

▲

torginus 2 hours ago | parent | prev | next [-]

> Ultramarine 1 Flash is Fable-class on knowledge work spanning finance, law, consulting, and operations. It is also 60x cheaper than Fable.

This is not a general purpose model, it makes sense to only bench it on tasks it's supposed to be good at, which is to help out finance-related knowledge work.

▲

jchook 5 hours ago | parent | prev | next [-]

The benchmarks are not ideal.

	▲	icantevenhold 26 minutes ago \| parent [-]
		When are they ever? I can’t remember a time when I found a benchmark to be genuinely useful even for deterministic software

▲

nurettin 2 hours ago | parent | prev | next [-]

This is not an aside at all. It is on point. It seems like whatever statistical trick they used also inflated gemini 3.8 to absurd scores as well.

▲

6 hours ago | parent | prev | next [-]

[deleted]

▲

yieldcrv 6 hours ago | parent | prev [-]

in my startup, our publicist used to make buzz feed style listicles, consistently listing us as the second or third solution

but it was way deeper than that, because it wasn't actually about people looking up some search string, it was actually about predictably pinging the google alerts for the top company and their founders, and then my same publicist messaging them as a rando on linkedin or email suggesting that they join us for a fireside chat at a conference they had their hands in

with the fireside chat in front of everyone, and the record of it, boosting us to the collective conscious riding off of the more established brand's presence

but yeah our personal google results also looked amazing for several years too

everything serves a parallel purpose, but yeah there's almost no utility in self aggrandizing on your own homepage