Man you Apple Fans really live under a rock.

Mini PC with Occulink, EGPU dock, 750w power supply and a refurbished 3090 is 2500. It can run Gemma4:31b at 1000+ tok/sec prefill, ~40 tok/sec generation (which is slow, but a fully connected model is way better for all tasks)

if you want more specialized partial activation models like Qwen, you can get 100+ tok/sec generation.

This beats anything that Apple has by a long shot. You don't need to run large models for usefulness. Large models can get things write the first time, but thats their only advantage. Running small models that are good enough like gemma4:31b fast allows you to do faster iterations.

And if you really want, you can also do multi GPU setups if you want slightly slower model loading with usb4/thunderbolt ports.

TLDR; local ai has already been useful for quite some time if you arent brainwashed into believing that Macs are the best for running local inference.

▲

Xeoncross a day ago | parent | next [-]

> "Man you Apple Fans really live under a rock... ...refurbished 3090 is 2500"

Before you make assumptions, there are valid reasons for the choice I made:

I paid $1k for a refurbished M1 Max 64GB macbook that can run Qwen 3.8 27B at 24tok/sec (looks like they are about $1.5k on ebay). In my case, Apple was a lot cheaper (used) if you're looking at price-per-token. Yes, it's almost twice as slow as a 3090, but it's also half the price /performance ratio.

Qwen 3.8 27B Q4 with 64k:

3090: ~40tok/sec

M1 Max : ~24tok/sec (MLX)

In fact, the Max can even do 265k context which isn't possible with a single 3090 last I checked (without dropping to Q3-Q2)

	▲	ActorNightly 19 hours ago \| parent [-]
		>Qwen 3.8 27B Q4 with 64k, 3090: ~40tok/sec Try more like 70-130 tok/sec. https://github.com/L-Forster/open-jet/tree/master/megakernel. And also Qwen models famously chase benchmarks. They are good for writing code, but fall behind in other areas. Gemma4:31B is 6gb larger and does a much better job across all areas. https://www.reddit.com/r/LocalLLaMA/comments/1vyzopv/gemma4_... Go ahead and run Gemma4 on your M1, and you will find that its unusable at 12 tok/sec. And this is also not touching prefill, which is the biggest thing that you need fast, and Apple sucks in this regard compared to nvidia cards. With gfx cards, you can throw big codebases at the local models and it will happily crunch through them in very little time. Sure, spending 1.5k on a Macbook enables you to run a local model, and if you really have a good reason for running local (like running heretic/abliterated models) for tasks you don't mind waiting for, it works. The difference is, spend 1k more and let you never have to pay for tokens, and you can run inference all day long without your computer thermal throttling, like your Macbook. And I don't need to make assumptions when you say idiotic stuff like "We're getting close to local AI being really useful", like you have any idea what you are talking about. On par with being an Apple fan though.

▲

labcomputer 16 hours ago | parent | prev | next [-]

A $2500 Mac Studio with M5 Max running Qwen3.8-27b or Gemma4:31b in oMLX gets about the same speeds you quoted... and it comes with a warranty and has enough memory not to choke at longer context lengths.

	▲	ActorNightly 5 hours ago \| parent [-]
		Qwen3.8-27b can be ran at 100 tok/sec average https://github.com/L-Forster/open-jet/tree/master/megakernel. Gemma4:31b, which is the better model with more complex architecture (and thus can't be compiled into a megakernel), is gonna be way slower in prefill, and ~50% slower than a 3090 even with oMLX. You can't get away from much slower shared ram vs vram, and less parallelism compared to 3090. The other advantage of an eGPU setup, you can simply add/swap performance. Dual 4070s give you 24b of ram and you can still get about 80% of the performance of a single 3090, and actually you get faster prefill due to more parallelism. So in terms of money/usefulness, Mac always loses.

▲

abujazar a day ago | parent | prev [-]

Most people don't run their own LLMs, let alone install GPUs themselves.

I'm one of those Mac users, but I do have a linux tower for remote dev work and LLMs.