I am not sure a best practice in one area translates nicely to another. LLM's power comes with it being a monolith.

9rx 2 hours ago | parent | next [-]

> LLM's power comes with it being a monolith.

Does it? All of the latest frontier models that have made technical details publicly available are using MoE; smaller specialized units that work together.

	▲	nixon_why69 12 minutes ago \| parent [-]
		Yeah but they're all trained together along with their gate tensor. Evaluating a moderate length prompt pulls in most/all of them.

▲

autuni 3 hours ago | parent | prev [-]

Does it though? Let's say you use an LLM to write code, do you think it benefits from the fact that the same model is capable of translating corporate messaging into internet slang for marketing?

I would assume a smaller model trained specifically to produce code based on natural language input would perform at least just as well, which would already be better given the smaller size / probably lower training resource requirements. This is of course, as already said, an assumption. But generalizing typically comes at a cost, there is no reason why this wouldn't apply to LLMs, it even applies to humans.

▲

IanCal 32 minutes ago | parent | next [-]

Or would it be worse as it wouldn’t have the broader knowledge to understand the reason you’re coding what you are anyway, understand the logs and iterate?

Coding an exercise app will benefit from a model that can code and understands what exercise is. A car showroom website will benefit from a model that understands what relevant filters would be.

	▲	autuni 2 minutes ago \| parent [-]
		Of course there are synergies between different fields. I'm not saying cutting everything off as close as possible to it's purpose must be the best way. But that's a long way from a single generalized model.

▲

heggy 2 hours ago | parent | prev [-]

Isn't this just the Bitter Lesson over and over again? Specialised approaches get outperformed by more compute and a simpler general approach pretty much every time.

	▲	try-working 43 minutes ago \| parent [-]
		Bitter lesson is wrong. Large general models are outperformed by small specialized models.