There are an absolutely stunning number of ways to lose a whole bunch of money very quickly if you're not careful renting compute.

$3,000 is well under many "oopsie billsies" from cloud providers.

And that's outside of the whole "I own it" side of the conversation, where things like latency, control, flexibility, & privacy are all compelling reasons to be willing to spend slightly more.

I still run quite a number of LLM services locally on hardware I bought mid-covid (right around 3k for a dual RTX3090 + 124gb system ram machine).

It's not that much more than you'd spend if you're building a gaming machine anyways, and the nifty thing about hardware I own is that it usually doesn't stop working at the 5 year mark. I have desktops from pre-2008 still running in my basement. 5 year amortization might have the cloud win, but the cloud stops winning long before most hardware dies. Just be careful about watts.

Personally - I don't think pi clusters really make much sense. I love them individually for certain things, and with a management plane like k8s, they're useful little devices to have around. But I definitely wouldn't plan to get good performance from 10 of them in a box. Much better off spending roughly the same money for a single large machine unless you're intentionally trying to learn.

▲

lumost 3 days ago | parent | next [-]

At the local/hobby scale, it’s very much a “do whatever” area. But I can rent a 4090 for a little under a dollar an hour, and I can rent a b200 for $6, it’s very hard to claim I’ll use 10k+ hours of gpu time on a b2000 I buy for myself.

	▲	horsawlarway 3 days ago \| parent [-]
		So 83 days for payback at the 2k sticker price for the 4090? Sounds like a good time to buy a 4090... Like, if you buy that card it can still be processing things for you a decade from now. Or you can get 3 months of rental time. --- And yes, there is definitely a point where renting makes more sense because the capital outlay becomes prohibitive, and you're not reasonably capable of consuming the full output of the hardware. But the cloud is a huge cash cow for a reason... You're paying exorbitant prices to rent compared to the cost of ownership.

▲

0xbadcafebee 3 days ago | parent | prev [-]

You could also spill a can of Mountain Dew over the $8,000 AI rig next to you. Oopsies can happen anywhere...

If it's for personal use, do whatever... there's nothing wrong with buying a $60,000 sports car if you get a lot of enjoyment out of driving it. (you could also lease if you want to trade up to the "faster model" next year) For business, renting (and managed hosting) makes more sense.

	▲	horsawlarway 11 hours ago \| parent [-]
		So you got downvoted already, but to clearly address this: If I spill something on my own hardware, the max out-of-pocket amount I lose is the amount I spent on that hardware. If I run up an AWS/GCP/Azure bill accidentally... the max out-of-pocket amount I lose is often literally unbounded. Are there some guardrails you can put around this? Sure. But they're often confusing, misleading, delayed, or riddled with "holes" which they don't catch. Ex - the literal best AWS offers you is delayed "billing alarms" which need to be manually enabled and configured, and even then don't cover all the services you might incur billing charges for. It's not that "Oopsies" can't happen locally - it's that even if they do, I have a clear understanding of the potential costs by default, and they're much less intangible than "I left a thing running overnight and I now I owe AWS a new car worth of cash". The worst case for a misconfigured bit of software locally is that my machine stalls and my services go down (ex - overloaded). The worst case for a misconfigured bit of software in AWS is literal bankruptcy. Think about that for a minute.