This is just temporary though, right?

With the benefit of LLMs already being proven, in a couple of years we will have vastly better hardware for inference I guess.

I feel like now hardware is stagnating a bit, because the software side has moved too fast for the hardware to catch up. Once we settle on sone good, optimal software architecture for the models, dedicated hardware will easily increase throughout by 10x or 100x, for a fraction of the most.

LLMs seems quite simple, maybe we'll be able to print at home our own chips with the models.

I think the path of least resistance will end up being the cheapest and that is scaling up the parameters a ridiculous amount until you get highly capable models that you can develop/distill/design the RAM efficient models. Going straight for low param is foolish and just a cope by smaller labs because they don't have the compute/talent to train the large ones.

Many on HN are in deep denial about this imo.

I'd assume the closed weight models are all working on shrinking their parameter counts anyways. They too benefit from smaller models. It'd be foolish for these SOTA labs to not be working at reducing parameter counts.

Let's not forget the Bitter Lesson. Small models sound really nice but at some point you're just fighting the laws of information theory. Efficiency gains on the small model side are nice, but efficiency gains + giant model tends to be even better...

I would say even without rampocalypse there would still be the strong incentive to innovate at the edge and under more extreme constraints. The incentives are just even stronger now.

I'm looking forward to seeing what types of new things people create over the coming years once there is less obsession with massive unwieldy LLMs. I think the incentives are just too strong to ignore.

I am literally betting my company on this being true.

I honestly hope to see this across all applications, games, services, operating systems, etc. We've been in a period of wasteful RAM usage for over a decade. Constraints, whatever their origin, can be a good thing.

I've seen ones that are configurable to pick a trade off point between lower cost (cache stickiness) and routing performance (best model for that turn).

But yeah I'm skeptical all this overhead is worth it.

Caching should be possible as long as all the models use the same shared cache. The models don't even need to be running on the same server if the shared cache is distributed.

I have a feeling people reading this are thinking that a model router would be used to route between different providers. And in that case, a shared cache would be impossible, although some caching would still be effective. I think, ideally, a router like this is in front of a set of models hosted in one place.

The repo is probably a better entrypoint to it, bit more concise description than the press releases: https://github.com/NVIDIA-NeMo/Switchyard (Notably: "Experimental software. Not for production use."). Unclear if they actually want you to deploy it or not, press release says yes, README says no, do with that what you will.

Doesn't seem to mention "cache" in the README nor the docs, but the code has mentions of it (https://github.com/search?q=repo%3ANVIDIA-NeMo%2FSwitchyard+...), I'm not sure what their thinking is there. "Good luck" essentially? Seems to be per-provider at best, but weird position for a routing library to take.

k

While I agree with the spirit, I don't think the solution to bad prose is slightly less bad prose. We can write good prose instead.

In my opinion: the only way forward is zero-knowledge-proof authenticated social media.

We can’t have legitimate debate if we have to assume a few bad actors are cloning their voice by the thousands, poisoning debate.

If we can pin one account to a real person, we won’t get rid of LLM-content and misinformation, but at least we can hold them accountable.

Not by the looks of it, but it got me thinking, currently in the middle of Level1Techs coverage on the model and switchyard and he mentions "how easy it is to customize it". Fully admitting that I haven't yet read the docs, my issue with that is "we can train LORAs for 35B as well, why use this (according to benchmarks) worse model for customization instead of a slightly bigger better one?"

Assuming I eat my words after going through the docs and this is actually a more efficient model / loras adapt better, I don't see as much value in it as is, as a REAP of it (remove least-important experts, domain-locked tests show ~98% retained accuracy) to something like 20B-A3B (rouhgly matching gpt oss, which while a good model, is outdated knowledge-wise and not as good with tool in my xp).

Having a 20B-A3B model at q4 that has a lora to be your local orchestrator (delegating coding to server/cloud models) and ci/cd runner does start sounding like an appealing proposition to me, as that would fit in 16gb vram easily (fitting many consumer gpus and 24gb macs).

Muse Glimmer 30B seems to be on par with Qwen 3.6 27B (4 months old)

but

Qwen 3.8 27B is dropping this week...

The top 4 models on that site are all variants of Gemini Flash? That does not match my experience at all.

lightning is sparse, glimmer is dense

and Glimmer has 10x active params of Lightning. Meaning ~ 10 slower on same HW

At what cost difference?