Neither is the right alternative to compare to. You aren’t going to hit 100% utilization (if you are, ignore me, this doesn’t some to you, and write a blogpost for me to read and share).

The comparison should be against renting in the cloud for the duration of your task for training and research or using pay-per-api-call providers for general inference instead of buying your own hardware (and paying the electricity and cooling bills on top), because let’s face it, the models you want to use are probably the same ones available on inference providers (but, yes, some are more trustworthy than others).

Speaking as someone that does ML/AI research, you are essentially paying a huge premium for being able to just run your Python script at any time without setting up a deployment script and harness to run the job remotely, while your hardware sits essentially idle the rest of the time.

The only way to make the math work is if you rent your hardware in the background for inference while you’re not using it in anger, but despite all the startups and promises that has never become as streamlined as mining bitcoins or shitcoins used to be and they don’t pay out as much as they say they would. Renting your hardware for training is another option but doing that is a lot more involved, options are fewer and farther in between, you won’t get as much utilization out of it, and doesn’t let you feasibly abort running tasks at a moment’s notice.

My card (RTX Pro 6000) is always doing something all the time from my queue; like some synthetic dataset generation up next. I still actively use runpods and openrouter for scaled stuff, I was spending a bit and then did the maths, and invested in it.

The maths to me was basically equivalent to prepaying for 242 days of runpod pricing for the same GPU; and I reckon I'd be able to get 6+ years of use out of this card with 96GB.

Plus there's the resell value -- it's actually appreciated by ~50% since I bought it.

Plus I do really enjoy that it's 100% local. I wouldn't feel comfortable giving my agents this much information if inference wasn't 100% local.

I wouldn't get another one, I wouldn't have as much value, but one is definitely paying off for me on the financial side.