Remix.run Logo
▲ Inkling 2 Mini(thinkingmachines.ai)
297 points by jasondavies 2 hours ago | 34 comments
▲montroser an hour ago | parent | next [-]

This is bonkers good if these benchmarks hold up in real-world scenarios. To be at the level of Qwen3.8-27B with a looped transformer model is quite the flex. They just left this note on reddit:

Hope that folks find this model more amenable to local usage - the sizing was largely informed by what folks on this subreddit ask for the most haha!

There are two particularly interesting aspects of this model for local users:

We do our training end-to-end in nvfp4, and although we haven't exhaustively benchmarked this, folks like Unsloth have noticed that our models seem to respond very well to being quantized, maintaining 90% top-1 even at at 1-bit quantization.

This model is looped, which allows more depth at the same parameter count, which we particularly notice seems helpful for coding tasks. :)

This is our first model targeted at local users, so let us know how this model is and anything you want to see out of future iterations!

https://www.reddit.com/r/LocalLLaMA/comments/1y1diuk/thinkin...

▲monksy 8 minutes ago | parent | prev | next [-]

The GGUF version of this that can run on Ollama/Llama.cpp is: https://huggingface.co/unsloth/Inkling-2-Mini-GGUF

▲stillpointlab 3 minutes ago | parent | prev | next [-]

I'm guessing one of these local model companies will get picked by Apple sooner or later. It is just so within their business model and they have the hardware.

▲yread 43 minutes ago | parent | prev | next [-]

Recently SensaNova also released a looping model with unbelievable numbers

https://www.sensetime.com/en/news/sensenova-loop-9b-release-...

▲2001zhaozhao 25 minutes ago | parent | prev | next [-]

I think this model will be a crazy big deal, it might be the first model that can reliably run multi-agent reasoning harnesses on local hardware, as this is a MoE model and can have several agents each with their own KV caches that fit into VRAM.

Dethroning Qwen 3.8 is a big deal too, and I like their section about CoT monitorability. (This model beats Qwen on benchmarks - TML is not known to benchmaxx nearly as hard as Qwen.) I guess TML really are a bunch of rationalists who know what they're doing, sort of like Anthropic.

▲WiSaGaN 6 minutes ago | parent | prev | next [-]

This is actually a big deal. I think probably by 2028 we will get a large number of daily tasks done reliably by laptop size local models.

▲rsolva an hour ago | parent | prev | next [-]

Going to test this right away on my AMD 7900XTX and see how it does compared to Qwen 3.8 27B (or DaisyRose 27B that I'm currently running). Given that Inkling 2 Mini is a MoE, it should provide way higher decode speed, but with comparable performance. Exiting!

▲AndrewKemendo 26 minutes ago | parent [-]

Please reply when you have tested it I’m curious.

Also do you know what the best local setup is?

▲jakswa 13 minutes ago | parent | next [-]

I had astra patch in both required llama.cpp PRs and it's passing a few tests locally on my 7900XTX so far, but ~500 tok/s prefill and ~26 tok/s decode are sobering: The new three-time-looping might be cutting speeds. Maybe there's something here still though, maybe it needs less reasoning output for results to be usable/good? More testing needed!

edit: found some oudated mesa/radv/shaderc/glslang packages to update, and up to 34 tok/s now. MORE MORE

▲rsolva 9 minutes ago | parent [-]

Hm, that seems strangely slow. Qwen3.8-27B/DaisyRose-27B Q4 currently run with ~66 t/s and ~800 t/s prefill with 3 slots and about 190k kv cache. It performs really well so far.

▲jakswa 7 minutes ago | parent [-]

Yeah I've had 27B (dense model) up to similar speeds on this setup as well. I'll hold off to compare my speeds to others, maybe it's too early or it's particular to my setup.

▲rsolva 16 minutes ago | parent | prev [-]

For 7900XTX you mean? I run Fedora Silverblue 44 at the moment and could share my current containerized setup with DaisyRose 27B.

▲gchadwick an hour ago | parent | prev | next [-]

Very interesting to see them publish a looped transformer. It's been reported that OpenAI are now using looped transformers, though of course not confirmed.

I think this may be the first public example of a looped transformer model good enough for real world use cases?

▲gok 21 minutes ago | parent | prev | next [-]

Very cool, and the also released a research version with variable looping support: https://huggingface.co/thinkingmachines/TML-d24-varloop-rese...

Idea being you can configure at run time how much thinking you let the model do per token.

▲mattnewton an hour ago | parent | prev | next [-]

Benchmarks look good, and inkling 2 small felt better than the benchmarks suggested, so I am very excited to play with this.

From a research perspective, I'm even more interested in the varlooped variant they're releasing https://thinkingmachines.ai/model-card/tml-d24-varloop-resea... I'm not aware of any other working open source models like that.

▲liuliu an hour ago | parent | next [-]

My current feedback on Inkling 2 Small: 1. it feels intelligent, and can solve open problems throw at it like other quasi-frontier models (DeepSeek 4 Flash 0731 / DeepSeek 4.1 Flash 1017 / GLM 5.3 Flash etc). 2. its responses sometimes can be odd (they are honest about it in the blog post!), can have wrong use of markdown syntax, sometimes overly brief, etc. 3. tool uses are good, but sometimes have some old take on tools (too quickly to assume image generation tools cannot do text etc).

EDIT: by "open problems", I meant the problems I don't really formulate how to solve it first in my mind and then throw to it. I just describe what I want and let it to figure it out.

▲kcb an hour ago | parent | prev [-]

Sounds like https://huggingface.co/nvidia/NVIDIA-Nemotron-Labs-Dory1.0-4...

▲xlayn 11 minutes ago | parent | prev | next [-]

When you are sure enough about what you are doing to include Qwen 3.8 27B in your compare.. we know you are talking... let's test drive it!

▲jakswa an hour ago | parent | prev | next [-]

The masses lying in wait for Qwen 4 cry out in joy. A savior? Could it be?!

▲jakswa an hour ago | parent | prev | next [-]

When inkling first released I tested it out and thought it was the best prose from an LLM in my quick sampling -- easy-reading and easy to understand. Big hopes I can write code with one of these models one day.

▲summarity an hour ago | parent [-]

Absolutely. Inkling for prose (and vision), Deepseek 10XX for code are the sweet spot right now (they swap places for these jobs on my rig)

▲neya an hour ago | parent | prev | next [-]

Wow, that coding demo was super impressive. Can't wait to try it out. Just need to wait for the hardware market to cool down a bit so we can run these on our own commodity hardware.

▲walrus01 an hour ago | parent | prev | next [-]

Looking at the link to quantizations it looks like unsloth has already quantized it:

https://huggingface.co/thinkingmachines/Inkling-2-Mini

https://huggingface.co/unsloth/Inkling-2-Mini-GGUF

In terms of the active parameters and token/s on slow or less capable hardware it seems like it should be in the same general performance category as qwen 3.6 35B A3B.

▲jakswa an hour ago | parent [-]

might need this if trying to test on llama.cpp https://github.com/ggml-org/llama.cpp/pull/32695 -- "Adds support for https://huggingface.co/thinkingmachines/Inkling-2-Mini - uses a looped architecture and this PR adds looped support"

▲hrncn 31 minutes ago | parent | next [-]

Let's hope these make it into master, and don't end up like the diffusiongemma patchsets. It is always a bit sad if the exotic and exciting models don't make it into master and all the people using prebuilt llama can't just give them a try. Tho I can understand why these in particular might not be wanted in master.

▲jakswa 43 minutes ago | parent | prev [-]

and this lol: https://github.com/ggml-org/llama.cpp/pull/32694

▲lenerdenator 23 minutes ago | parent | prev | next [-]

This looks like it could run on my M2 32GB MBP.

I honestly would rather have nice, focused-purpose local models than huge "let's make AGI" datacenter-hosted models. Will have to try it tonight when I get home.

▲AndrewKemendo 27 minutes ago | parent | prev | next [-]

Can anyone running one of these locally give any sense of how good they are?

▲holistio an hour ago | parent | prev | next [-]

I'm starting to believe Gestalt is actually theirs. This is very nice.

▲christkv an hour ago | parent | prev | next [-]

I'm loving this trend. My framework desktop is looking like a better and better investment.

▲Iolaum 41 minutes ago | parent | next [-]

Indeed. I 've been using Sage A1.5 35b-A3b to have more speed because of strix halo limited memory bandwidth. Looking forward to testing Inkling-2-mini once it has lamma.cpp support. Genuinely excited from a model outside Qwen3.6-35B-A3B ecosystem competing in that space.

P.S. Qwen3.8-Flash-Next is also a very good model for Srix-Halo/128Gb Framework Desktop but I liked to have more RAM for other things as well even when I was running it in Q4. Usable though.

▲rapind 43 minutes ago | parent | prev | next [-]

What are you running on it? DSF 4.1?

▲topaztee an hour ago | parent | prev [-]

tell me more about this framework desktop..

▲lwarfield 32 minutes ago | parent | prev [-]

Omg I've been working on the same thing! I was doing this off a hunch from David Noel ng's RYS posts!

https://dnhkng.github.io/posts/rys/