That video seems to show it takes ~4 seconds to process 43 input tokens. Considering the amount of tokens coding-agents chew through, even when just exploring and not even implementing anything, feels far away from "acceptably fast". Say one file is 500 tokens (small file), it'd take ~a minute just to process it?

You mention "See later in the reply for the prefill at ~300 t/s" but I didn't see that in the video nor in any of the Twitter replies, could you link where that is supposed to be happening? Is that warm context perchance?

▲

zozbot234 3 hours ago | parent | next [-]

43 tokens is a tiny input context when compared with anything agentic. Prefill on SSD streaming is proportionally much faster with larger contexts, because expert loads can be reused across tokens and the engine can also opt to preload entire model layers into memory, without waiting for the outcome of expert routing.

▲

cyanydeez 4 hours ago | parent | prev [-]

all my local models do just fine (128GB Strix Halo, 72GB Blackwell) loading data. I'm not sure you were sold the right bill of goods for "agentic" work. I keep these things busy via mind power for a few hours, then get exhausted.

So I implenented a task queue so when I do have all the ideas, I can just set them to work, and "chew" through the work.

I get that you can concievably get 100 agents to do something interesting, but 99% of my work doesn't seem to need to divine naiver stokes.

Other than that, you might just have a skill issue if you rely so heavily of spinning up a bunch of agents to tell you what the next TODO item is.

	▲	zozbot234 3 hours ago \| parent \| next [-]
		Ironically, local models are much more efficient when running multiple parallel agents, provided that KV storage requirements are not excessive (not an issue with DeepSeek 4.1 Flash). It does not make any individual agent faster, but it does raise overall throughput quite a bit. The limit is then set by compute constraints rather than memory bandwidth, so something like DGX Spark or AMD Strix/Gorgon Halo can be expected to do better than the Apple-based alternative.
	▲	embedding-shape an hour ago \| parent \| prev [-]
		> 128GB Strix Halo, 72GB Blackwell Lol yeah of course, that's not the Apple hardware showcased in the video. Sure, my 6000 Pro also does 11K in prompt processing, but my comment wasn't generally about local models, I thought that was obvious. > I get that you can concievably get 100 agents to do something interesting Uuh, OK, cool. Let those vibesloppers do that process if they'd like to, I think most professional software developers already found that process to produce slop and not production-worthy software, so we continue with our process that produce code of quality instead of aiming to produce as much as possible. > Other than that, you might just have a skill issue if you rely so heavily of spinning up a bunch of agents to tell you what the next TODO item is. Huh? Where are you getting this from? Where did I say anything of the sorts?