But in the end, isn't this the same idea with the MoE?

Where we have more specialized "jobs", which the model is actually trained for.

I think the main difference with agents swarm is the ability to run them in parallel. I don't see how this adds much compared to simply sending multiple API calls in parallel with your desired tasks. I guess the only difference is that you let the AI decide how to split those requests and what each task should be.

▲

zozbot234 3 hours ago | parent [-]

Nope. MoE is strictly about model parameter sparsity. Agents are about running multiple small-scale tasks in parallel and aggregating the results for further processing - it saves a lot of context length compared to having it all in a single session, and context length has quadratic compute overhead so this matters. You can have both.

One positive side effect of this is that if subagent tasks can be dispatched to cheaper and more efficient edge-inference hardware that can be deployed at scale (think nVidia Jetsons or even Apple Macs or AMD APU's) even though it might be highly limited in what can fit on the single node, then complex coding tasks ultimately become a lot cheaper per token than generic chat.

▲

XCSme 3 hours ago | parent [-]

Yes, I know you can have both.

My point was that this is just a different way of creating specialised task solvers, the same as with MoE.

And, as you said, with MoE it's about the model itself, and it's done at training level so that's not something we can easily do ourselves.

But with agent swarm, isn't it simply splitting a task in multiple sub-tasks and sending each one in a different API call? So this can be done with any of the previous models too, only that the user has to manually define those tasks/contexts for each query.

Or is this at a much more granular level than this, which would not be feasible to be done by hand?

I was already doing this in n8n, creating different agents with different system prompts for different tasks. I am not sure if automating this (with swarm) would work well in my most cases, I don't see how this fully complements Tools or Skills

▲

zozbot234 3 hours ago | parent [-]

MoE has nothing whatsoever to do with specialized task solvers. It always operates per token within a single task, you can think of it perhaps as a kind of learned "attention" for model parameters as opposed to context data.

	▲	XCSme 2 hours ago \| parent [-]
		Yes, specific weights/parameters have be trained to solve specific tasks (trained on different data). Or did I misunderstand the concept of MoE, and it's not about having specific parts of the model (parameters) do better on specific input contexts?