| ▲ | nik736 3 hours ago | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
"Decision 1.1 is the latest decision model from Microsoft, 50x faster and 79x cheaper than GPT-6.1 Sol"... uhm okay. What a useless comparison. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| ▲ | jmoore15 13 minutes ago | parent | next [-] | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Hi, John here from the team that built Microsoft Decision 1 and 1.1. We compare ourselves to other decision models in our blog post on Microsoft's Command Line blog from November 17 [1]. We name all of the public benchmarks we evaluated against in the post, and provide a breakdown by category. We also are on the Artificial Analysis leaderboard, which is an independent third party assessment. On AA, we score 68, above Gemini Decision 1, GPT-6 Luna Decisions, and Jev 1.14. [2] Our previous MD1 model from October is also benchmarked on JevBench, another independent benchmark for decision models. [3] On JevBench, we are 9th place for API models, ahead of Gemini Decision 1 and Jev 1.14. We look forward to seeing how the new MD1.1 scores on JevBench. With regards to cost: Our models are the same price as Jev on a per-token basis. However, on both JevBench and Artificial Analysis benchmarks, our models use fewer tokens per task than Jev. For example, our cost score for MD1.1 on Artificial Analysis at $140 per 1M tasks compared to Jev's $156 and Gemini's $345. There are similar stats for latency, where our new MD1.1 model is 2x faster than our previous one. [1] https://commandline.microsoft.com/microsoft-decision-model-g... [2] https://artificialanalysis.ai/evaluations/artificial-analysi... [3] https://benchmarkheaven.com/jev-models/api | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| ▲ | voidfunc 3 hours ago | parent | prev | next [-] | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
They're talking to executives not developers. I had this thought yesterday, the era of developers picking tech is probably over. Microsoft's wheelhouse is selling to upstairs. They're going to eat everybody's lunch that is still playing the old game of selling to the devs. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| ▲ | jackbravo 2 hours ago | parent | prev | next [-] | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
A little bit further down the page: > Decision 1.1 provides model routing integrated right into your applications so that the bigger, pricier models are only used when you really need them. On the AA Decision Index Microsoft is ahead of Jev, Google and OpenAI, while being faster and cheaper. Used by 4,000 customers, it’s 50x faster and 79x cheaper than GPT-6.1 Sol. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| ▲ | jrflo 3 hours ago | parent | prev | next [-] | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Makes me doubt the rest of their claims when you see something like that | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| ▲ | andai 2 hours ago | parent | prev | next [-] | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Our banana is 9 times yellower than the competitor's oranges. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| ▲ | peterbell_nyc 2 hours ago | parent | prev | next [-] | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
It is also substantially better on most frontier benchmarks than a bash script with a random number generator. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| ▲ | dontknowmuch 2 hours ago | parent | prev | next [-] | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Yeah...they really should be comparing that to Jev. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| ▲ | orthogonal_cube 3 hours ago | parent | prev [-] | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Tangential to this, but I really hate the use of “#x times cheaper” when describing a price difference, especially when it takes place after an “#x times faster” statement. It’s initially confusing and makes it harder to mentally calculate the actual difference. Just give it as a percentage! | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||