The model-as-a-service clause (which is new to the K3 series) actually only requires $20 million over 12 months - i.e. yearly revenue.

Just one of our analysts was burning around 30-40 million tokens per project.

EDIT: per project, not per month

EDIT 2: digging out an old bill from 8 months ago when we bought the server, $14k per month is what we were running and it was going up, a lot.

Anthropic is very good at either convincing or forcing (I’m not sure which) companies to pay for tokens rather than for subscriptions. My old job moved from subs to tokens and everyone started running out of budget mid month. It was a bad move for a company that laid off all of QA and 2/3 of Engineering and committed to everyone making up for it with LLMs.

Oh boy... we are talking about insane token spends in corps at the moment. 100$ is a blip.

I think you should compare with api cost? I am using $200 plan and it is probably around $12k in token cost....

I didn't dig into Kimi's MaaS legalese, but i dont really understand how that knee-high fence is supposed to work long term.

All the service provider needs to do is provide the option to rent out VM's with access to the inference cards.

Have the customer provide the model.

Maybe provide a bit of a speedup, by letting the customer mount the model by hash.

Sure, that all sounded way too technical 10 years ago, and somebody would have said that's already too much annoyance to deal with for average customers.

Reluctantly i'd agree.

That was 10 years ago. Nowadays, with some proper docs, this entire set up can be up and running in 5 minutes.

Maybe for the API rate, but for OpenCode Go they're regularly making deals to get users a lot more than their money's worth on open source models.

All these very large models are sparse mixtures of experts (SMoE), where typically only 2-3% of weights are active for a given forward pass.

Usually there are some static experts in each attention layer, plus a whole bunch of dynamically-routed experts. Typically some form of LRU cache is used for the dynamic experts.

So you can keep only the static expert(s) and routing layers in memory and then stream the dynamic experts from SSD, at a modest inference speed cost driven by the latency required to load layers from SSD when they aren’t already in the memory cache.

The LRU cache works reasonably well because a given task will often hit the same experts on each iteration of the forward pass.

For example, on a Mac Studio, Qwen3.8-flash-next (180b total parameters) with 50% of experts in memory and n-gram offloading has about 1/4 to 1/3 the inference speed of everything in memory. That requires 32GB of vRAM for the weights (plus a little extra for context, obviously)

It's not based on loading the full 1.5TB model into VRAM or system RAM, it's just streaming parameters on demand from SSD - which is worthwhile since the MoE portion of the model is quite sparsely accessed. This has been demoed on a 128GB M5 MacBook Pro with lousy performance (0.3 tok/s or so) because it had to load at least ~25GB from SSD per token (comprising the entirety of the sparsely routed active params) and potentially even reload some of the dense weights, depending on what exactly managed to fit on that platform. With 192GB you'd avoid any reloading for the dense weights and even cache a few layer-experts.

it's a mixture of experts so it's somewhat viable to stream weights from nvme

Last I checked AWS still sells EC2 instances using 2018 era GPUs. Most people underestimate just how much long tail demand there is and how long these cards last, despite the higher power consumption.

I agree, but it would have to be a place with plenty of energy and the only such place outside the US is a war zone.

Perhaps one option is building data centers along with solar power plants in sunny countries that have a lot of space. I'm not sure about the water/cooling situation though.

It’s called pacing the frontier

[flagged]

The rise of China due to hand-over-fist ceeding of power and influence fueled in equal parts by the right and left is so astonishing it'll certainly go down in the history books (written in Mandarin, of course).

Bonus points when the morons are doing so because they get all their political news from a Chinese platform. The [Americans] have gone insane.