To put this on perspective, consider that with 128GB of unified RAM it can run even fast: https://x.com/antirez/status/2119392658661749175 Like, coding-agent acceptably fast. See later in the reply for the prefill at ~300 t/s but can go even faster for larger prefills.

▲

sureglymop 3 hours ago | parent | next [-]

This seems great. I have 128GB of unified RAM and a AMD Ryzen AI Max+ 395 that I can use but I feel I'm still not using it as well as I could...

The main/best setup I have running so far is qwen flash next with halogen-server.

But if anyone knows if and how one can run even more impressive stuff, please let me know!

	▲	rpdillon 2 hours ago \| parent [-]
		I use lemonade llama.cpp with a 2-bit quant of DeepseekV4F 0731 that runs at about 15 tokens a second. I use it as my "smol" model in omp, with default going to OpenCode hosted Deepseek (faster inference). I haven't done work lately on Dwarf Star. I got it running a few weeks ago and had my agent make a couple of modifications to it to get it running, but it wasn't materially better than my lemonade setup. I might revisit after this post, though! https://github.com/lemonade-sdk/llamacpp-rocm

▲

boricj an hour ago | parent | prev | next [-]

I've recently set up Ollama on my homelab server, which is a Topton N17 micro-ATX motherboard with a Ryzen 7 7840HS and 64 GiB of DDR5 RAM. It's no AMD Strix Halo, but it's a laptop CPU with a fairly beefy GPU and a decent amount of memory bandwidth.

I've had decent results with gemma4:26b (although it's more like batch processing when used as a coding agent) and my experience so far is globally in line with the first two sections of the ARCHITECTURE.md: fast with small contexts but performance degrades fast as its size increases. If the rest of the contents of that document are true, I'll need to retune my setup from scratch.

	▲	kennywinker an hour ago \| parent [-]
		Try strata. I’m not sure how well it handles igpus but if it does i’m pretty sure that system will go screaming fast

▲

embedding-shape 4 hours ago | parent | prev [-]

That video seems to show it takes ~4 seconds to process 43 input tokens. Considering the amount of tokens coding-agents chew through, even when just exploring and not even implementing anything, feels far away from "acceptably fast". Say one file is 500 tokens (small file), it'd take ~a minute just to process it?

You mention "See later in the reply for the prefill at ~300 t/s" but I didn't see that in the video nor in any of the Twitter replies, could you link where that is supposed to be happening? Is that warm context perchance?

▲

zozbot234 2 hours ago | parent | next [-]

43 tokens is a tiny input context when compared with anything agentic. Prefill on SSD streaming is proportionally much faster with larger contexts, because expert loads can be reused across tokens and the engine can also opt to preload entire model layers into memory, without waiting for the outcome of expert routing.

▲

cyanydeez 3 hours ago | parent | prev [-]

all my local models do just fine (128GB Strix Halo, 72GB Blackwell) loading data. I'm not sure you were sold the right bill of goods for "agentic" work. I keep these things busy via mind power for a few hours, then get exhausted.

So I implenented a task queue so when I do have all the ideas, I can just set them to work, and "chew" through the work.

I get that you can concievably get 100 agents to do something interesting, but 99% of my work doesn't seem to need to divine naiver stokes.

Other than that, you might just have a skill issue if you rely so heavily of spinning up a bunch of agents to tell you what the next TODO item is.

	▲	zozbot234 2 hours ago \| parent \| next [-]
		Ironically, local models are much more efficient when running multiple parallel agents, provided that KV storage requirements are not excessive (not an issue with DeepSeek 4.1 Flash). It does not make any individual agent faster, but it does raise overall throughput quite a bit. The limit is then set by compute constraints rather than memory bandwidth, so something like DGX Spark or AMD Strix/Gorgon Halo can be expected to do better than the Apple-based alternative.
	▲	embedding-shape 19 minutes ago \| parent \| prev [-]
		> 128GB Strix Halo, 72GB Blackwell Lol yeah of course, that's not the Apple hardware showcased in the video. Sure, my 6000 Pro also does 11K in prompt processing, but my comment wasn't generally about local models, I thought that was obvious. > I get that you can concievably get 100 agents to do something interesting Uuh, OK, cool. Let those vibesloppers do that process if they'd like to, I think most professional software developers already found that process to produce slop and not production-worthy software, so we continue with our process that produce code of quality instead of aiming to produce as much as possible. > Other than that, you might just have a skill issue if you rely so heavily of spinning up a bunch of agents to tell you what the next TODO item is. Huh? Where are you getting this from? Where did I say anything of the sorts?