This is where I’m at. While I’d LOVE to have a powerful near-frontier model at home, I don’t have the extra funds necessary to purchase a tricked-out Mac Studio…and if I did, I’d pay off debts.

▲

7 hours ago | parent | next [-]

[deleted]

▲

datakan 7 hours ago | parent | prev [-]

Probably wouldn't matter if you had the funds either since MacStudios are 3-4 months out for purchase.

▲

netruk44 7 hours ago | parent [-]

And also the weights for GLM 5.2 are 1.5 TB. The only Studios Apple sells today are hard-capped to 64 GB (M4 Max) or 96 GB (M3 Ultra) only.

You'd need at least 24 new M4 Max Studios, or 16 new M3 Ultra Studios, or 3 used 512 GB M3 Ultra Studios just to power one GLM 5.2 instance. And even then, you're probably looking at < 5 tokens per second.

Personally, I think it makes way more sense to pay a model provider $3/1M tokens.

	▲	m_ke 6 hours ago \| parent [-]
		Deep Learning models are designed to get max throughput on GPUs, which ends up being batched workloads. You'll never get proper price competitive utilization on personal hardware vs a cloud inference provider that can batch and pipeline requests optimally to maximize utilization, unless you yourself start running batch jobs. Even once local hardware and models catch up to todays frontiers, by that time there will be 10x better cluster sized models available at a similar discount.