| ▲ | kbr- 5 hours ago |
| 200+ tok/s on local infra? That's crazy. How much did you pay for your hardware? |
|
| ▲ | crossroadsguy 4 hours ago | parent | next [-] |
| I find these local infra costs crazy heavy duty. I always assume these folks are getting their values worth or that infra is serving multiple people's requests at and justifying the cost. Like I once thought of buying a 48-64 GB RAM laptop (not even clsoe to what these models need) and I realised the equivalent cost will serve me for years of ≈ $30 per month API/sub access. So I assume the usage is also heavy duty, or if nothing then for privacy and safety and then I guess the price, if one can pay, will totally be worth it. |
| |
| ▲ | origini 2 hours ago | parent | next [-] | | You just need to imagine someone making big bucks in tech + being a little bit money ignorant, having a big house and a nice basement and cheap energy prices. Than it just makes sense that they already do this. But to be fair, if i move, enough space for battery, solar and a basement which can take proper ML hardware + heat exchanger, is in my future vision. | |
| ▲ | mrheosuper 4 hours ago | parent | prev | next [-] | | > I realised the equivalent cost will serve me for years of ≈ $30 per month API/sub access The advantage is no more 5-hours or weekly session limit anymore. Also no chance your proprietary code ends up in someone repo. | | |
| ▲ | amunozo 2 hours ago | parent | next [-] | | I still think only privacy is relevant to run on your own infra, never cost. | | | |
| ▲ | crossroadsguy 2 hours ago | parent | prev | next [-] | | That is literally my next sentence :D | |
| ▲ | david-gpu 2 hours ago | parent | prev [-] | | > The advantage is no more 5-hours or weekly session limit anymore That is not a concern with pay-as-you-go models like DeepSeek. It is stupid inexpensive. |
| |
| ▲ | lnenad an hour ago | parent | prev [-] | | If the only variable is money, local models will never make any sense at these prices of infra/subscriptions. For a lot of people, there are other variables though. |
|
|
| ▲ | swiftcoder 4 hours ago | parent | prev [-] |
| This has to be a distilled model? I don't see any route to 200 tokens/s without serious datacenter hardware |
| |
| ▲ | naraic0o 3 hours ago | parent [-] | | you mean quantized? it’s only got 16B active parameters so it can be pretty fast at decode. | | |
|
