Remix.run Logo
▲ nik736 3 hours ago

"Decision 1.1 is the latest decision model from Microsoft, 50x faster and 79x cheaper than GPT-6.1 Sol"... uhm okay. What a useless comparison.

▲jmoore15 13 minutes ago | parent | next [-]

Hi,

John here from the team that built Microsoft Decision 1 and 1.1.

We compare ourselves to other decision models in our blog post on Microsoft's Command Line blog from November 17 [1]. We name all of the public benchmarks we evaluated against in the post, and provide a breakdown by category.

We also are on the Artificial Analysis leaderboard, which is an independent third party assessment. On AA, we score 68, above Gemini Decision 1, GPT-6 Luna Decisions, and Jev 1.14. [2]

Our previous MD1 model from October is also benchmarked on JevBench, another independent benchmark for decision models. [3] On JevBench, we are 9th place for API models, ahead of Gemini Decision 1 and Jev 1.14. We look forward to seeing how the new MD1.1 scores on JevBench.

With regards to cost: Our models are the same price as Jev on a per-token basis. However, on both JevBench and Artificial Analysis benchmarks, our models use fewer tokens per task than Jev. For example, our cost score for MD1.1 on Artificial Analysis at $140 per 1M tasks compared to Jev's $156 and Gemini's $345. There are similar stats for latency, where our new MD1.1 model is 2x faster than our previous one.

[1] https://commandline.microsoft.com/microsoft-decision-model-g... [2] https://artificialanalysis.ai/evaluations/artificial-analysi... [3] https://benchmarkheaven.com/jev-models/api

▲voidfunc 3 hours ago | parent | prev | next [-]

They're talking to executives not developers.

I had this thought yesterday, the era of developers picking tech is probably over. Microsoft's wheelhouse is selling to upstairs.

They're going to eat everybody's lunch that is still playing the old game of selling to the devs.

▲samtp 2 hours ago | parent | next [-]

> the era of developers picking tech is probably over

Don't really know when that era ever existed

▲jerjerjer 2 hours ago | parent [-]

It still exists, just not in time, but rather at the company size.

▲mega-man 3 hours ago | parent | prev | next [-]

Has this been true pre-AI? It has always been sold to executives, no?

▲basch 2 hours ago | parent [-]

It was a short era

▲reynoldsalec 3 hours ago | parent | prev | next [-]

I'd be curious to hear how AI purchasing is really significantly different from other tech purchasing decisions. Seems like the enterprises that are accustomed to strong-handed mandates from on-high will continue to make tech decisions from the corner office, and the smaller orgs or more heterodox ones will continue to listen to ICs and management who surface new tools.

▲siva7 2 hours ago | parent | prev | next [-]

Even if they are talking to execs this comparison is still maliciously misleading. If facts and truth don't matter anymore anything in this age then why not add a few more zeroes to this number...

▲Avicebron 2 hours ago | parent [-]

It can't fail to meet expectations immediately, only after it's been sold to execs, they present to the board saying how technically savvy and fiscally responsible (compared to the dang engineers asking for claude), and then once it's worked into the ecosystem it can slowly become more well known how bad it is. The art of the deal as it were.

▲walrus01 2 hours ago | parent | prev [-]

> Microsoft's wheelhouse is selling to upstairs.

It's the modern era version of "Nobody ever got fired for buying IBM"

▲jackbravo 2 hours ago | parent | prev | next [-]

A little bit further down the page:

> Decision 1.1 provides model routing integrated right into your applications so that the bigger, pricier models are only used when you really need them. On the AA Decision Index Microsoft is ahead of Jev, Google and OpenAI, while being faster and cheaper. Used by 4,000 customers, it’s 50x faster and 79x cheaper than GPT-6.1 Sol.

▲jrflo 3 hours ago | parent | prev | next [-]

Makes me doubt the rest of their claims when you see something like that

▲matt_heimer 2 hours ago | parent [-]

Why? Look at their list of models again, each is specialized to a specific task. None of them are general purpose. It's not surprising that a task specific model can be better optimized for its task than a general purpose model.

▲jrflo 2 hours ago | parent [-]

They are acting like Sol 6.1, a general purpose model, is comparable to a decisions model. OpenAI has a decisions model, so they should be comparing to that.

▲andai 2 hours ago | parent | prev | next [-]

Our banana is 9 times yellower than the competitor's oranges.

▲peterbell_nyc 2 hours ago | parent | prev | next [-]

It is also substantially better on most frontier benchmarks than a bash script with a random number generator.

▲dontknowmuch 2 hours ago | parent | prev | next [-]

Yeah...they really should be comparing that to Jev.

▲orthogonal_cube 3 hours ago | parent | prev [-]

Tangential to this, but I really hate the use of “#x times cheaper” when describing a price difference, especially when it takes place after an “#x times faster” statement. It’s initially confusing and makes it harder to mentally calculate the actual difference. Just give it as a percentage!

▲flaunf221 3 hours ago | parent [-]

What do you mean "the actual difference"? 79 times cheaper means costing 1/79 of compared price. That's what it is. Or do you mean you want to see it in the format of decimal fraction?

▲orthogonal_cube 3 hours ago | parent [-]

Instead of “79x cheaper” say “1.26% of the cost” so readers can quickly do a mental cost comparison

▲flaunf221 an hour ago | parent [-]

Which comparison is the question.

"My budget gives me 100 million tokens for GTP 6.1 SOL. How many tokens for this new model would I get? Let me quickly divide by 0.0126 in my mind to figure it out."

x times more/less is the simplest form of comparison because bigger value is divisible by lesser value. I'm saying this because we first teach children division and multiplication and only then fractions.