| ▲ | hedora 19 hours ago |
| Without knowing which dots are which, this is useless. Let’s say I want to run qwen 27B or flash next on a machine I own, off solar panels that are also sunk cost. How did they price the marginal cost of a token? Depreciation on the box + reverse billing opportunity cost with the power company? |
|
| ▲ | wmf 18 hours ago | parent | next [-] |
| Luna is $20/month. That's cheaper than depreciation of any hardware. |
| |
| ▲ | monster_truck 17 hours ago | parent [-] | | Not that it actually matters here, but my last generation flagship gpu is currently listed on newegg for exactly twice as much as the launch MSRP that I paid. Even decent-enough/established ebay listings for used ones are +$500 lol | | |
| ▲ | hedora 7 hours ago | parent [-] | | Come to think of it, my Qwen box has gone up in value at a rate of $235 / month since I bought it. I guess I could sell it + it would retroactively pay for my Anthropic subscription. |
|
|
|
| ▲ | jimbokun 19 hours ago | parent | prev [-] |
| This article isn’t about that. It’s about comparing hosted models. |
| |
| ▲ | layoric 19 hours ago | parent [-] | | IMO then it is missing the forest for the trees regarding open weight models. Qwen 3.8 Flash Next or 27B, for example, are accessible enough to run at home/office at very usable speeds. Provided memory companies don't collude to keep margins at 80-90% they are at the moment, it seems to me this will become a lot more accessible for many use cases while models at this size still have room to improve. | | |
| ▲ | jimbokun 18 hours ago | parent [-] | | Except it’s very difficult for home users to afford the GPUs needed to run them with acceptable performance. | | |
| ▲ | notenchange 15 hours ago | parent [-] | | I have real world data here.
On a 5060 Ti (16GB) with 64GB RAM, I’m getting ~65 tok/s on decoding and ~450 tok/s prefil for Qwen 3.8 Flash Next. 256K context. This is totally suitable (speed wise) for coding. This is not some top of the line hardware (might be now, but it should be run of the mill for people that built their system <1.5 years ago). | | |
| ▲ | ozgrakkurt 14 hours ago | parent [-] | | How do you even fit that model in there? What quant is it? Are you using llama.cpp? | | |
| ▲ | notenchange 13 hours ago | parent [-] | | No, I'm using strata. This is a MoE model, so it keeps the last used model in VRAM, and the rest in RAM. If it needs, it swaps the less used model in VRAM for the model in RAM for the next token. Seems very slow, but it's actually fine. | | |
|
|
|
|
|
