This is essentially what Anthropic does with the advisor mode and what GitHub does with their multi-model pipeline using the HydraFusion model, which is just a large set of different models with different qualities used in a pipeline to produce an answer.

What I don't understand is how they all make it work with caching in general. Looking at how the advisor mode in claude work you seem to only be billed the output tokens and cache hit on the whole session when it calls out to the advisor, making it super cheap compared to running the more expensive model for the entire session.

▲

jeremyjjj a day ago | parent | next [-]

Pretty sure on advisor it needs to send the entire context to another model. The cost is mitigated a bit because there would be few output tokens from the advisor model normally.

Orchestrator seems to automatically delegate things to subagents as required is my read of it. This should be fine for caching.

	▲	onlyrealcuzzo a day ago \| parent \| next [-]
		> This should be fine for caching. When launching a swarm of smaller models, they can theoretically share cache amongst themselves, but not with the larger model (that already has it cached). They have different architectures. Further, it wouldn't really make sense for them to share the same cache anyway! The swarm, that is, they'd be working on independent things...
	▲	vegardx 13 hours ago \| parent \| prev [-]
		Yeah, the advisor mode does send the whole context, but you're only billed the cache input price for that, so either they just eat the cost, or they actually have something that makes it possible to move the cache between different models. I have absolutely no clue how these things works, but could it be that all the newer models are actually just the same model, with looped transformers, so it's just about how many passes it does internally that decides if it's Haiku / Sonnet / Opus / Fable? I might have misread the article a little bit, as you say, this is just workflow orchestration with subagents. Anthropic seems to think that this isn't the best approach anymore with their recent posts around subagent use and with the experimental advisor mode. So who knows.

▲

lemonish97 a day ago | parent | prev | next [-]

I've found hydrafusion to be great, although it sometimes does an unnecessary review pass at the end for simple tasks too.

▲

mark_l_watson 15 hours ago | parent | prev | next [-]

I would hope that a good harness would not just hoover up a large codebase into the context. Rather use some form of RAG (or similar) tech tuned for chunking, indexing, and searching code.

▲

rsalus a day ago | parent | prev [-]

Yeah, like how is it not constantly compacting when routing between models?