But, what model are you using?

and what hardware are you using?

yeah, on a 96GB Mac Studio and Gemma+Qwen, it's definitely fully doable. fully doable but not really for coding on 16GB. but svelter models and cheaper (eventually) hardware are coming!

▲

nezuzen 8 hours ago | parent | next [-]

"cheaper (eventually) hardware" Best case 2-3 years from now. Otherwise it will take a major global recession to get us anywhere near last year's prices.

▲

marcus_holmes 5 hours ago | parent | prev | next [-]

Macs are expensive hardware, but I'm always seeing people running LLMs on them. Is anyone running on cheaper generic hardware and Linux?

▲

brucehoult 5 hours ago | parent [-]

A Mac is cheaper than a high end GPU with the same amount of RAM.

▲

fsuts an hour ago | parent | next [-]

And use less power

▲

marcus_holmes 4 hours ago | parent | prev [-]

ah, right, so it's about Apple Silicon being fast enough to use instead of a GPU?

	▲	brucehoult 3 hours ago \| parent [-]
		They use the GPU but an Apple Silicon GPU has the same high speed access to all the RAM on the machine as the CPU does, rather than having its own walled-off maybe 16 GB VRAM in mainstream gaming GPUs or 24 GB in RTX 4090 or RTX 5090 (MSRP $1999 but in practice $3000-$4000 at the moment). Nvidia A100 (80GB VRAM) apparently cost $15,000 or so. Not only does Apple's unified memory give the GPU more RAM to use, but it also eliminates copying things between CPU RAM and GPU RAM. A Mac Mini with 48 GB RAM costs $1799. A Mac Studio with 96 GB RAM is $3999 — until March you could get a Mac Studio with 512 GB RAM for $3999, all of which could be used for your AI model. https://www.tomshardware.com/tech-industry/apple-pulls-512-m... Some are coming up used at silly prices. https://www.trademe.co.nz/a/marketplace/computers/desktops/a... NB NZ$44,999 is "only" US$25,772.

▲

Gigachad 6 hours ago | parent | prev | next [-]

I suspect hosted and local will converge when hardware prices come down and API prices go up. The massive rate of datacenter build out will be unsustainable. Right now the hosted models are massively cheaper than buying the hardware and running it yourself which signals that hosted is very subsidized.

▲

fluidcruft 6 hours ago | parent | prev [-]

If you don't have that hardware thr math of buying a depreciating computer is challenging if you are satisfied with the $100/month plans ($1200/year). A 96GB Mac Studio is ~$4k. I think if you have the hardware already as a sunk cost then yes it makes sense. But I'm not sure it is worth spending $4k for today's hardware vs waiting for newer hardware in a few years.