Could you say more about why it’s not viable for cloud inference? I’d love to understand this

Think about it from an ops perspective: for running inference, a model is essentially a gigantic binary blob that needs to be loaded into ram to process the request.

Updating weights means your changing this model.

To facilitate this feature, the inference provider would need to have your personalized model loaded into ram to process your request in a timely manner. Now multiply that by the number of users they're serving.

You can try to optimize it a bit by essentially gluing the changing weights as a "overlay" to be applied to the original model, but that comes with it's own trade-offs (aside from the fact it hasn't been implemented, just theoretically doable). And it would be seriously hard to implement well.

The amount of resources this would bind makes this infeasable for cloud inference like anthropic provides.

▲

stevenhuang 6 hours ago | parent [-]

Doesn't sound like you've been following the space. They exist, it's called LoRas. The Unsloth project is huge and whole purpose is to make it easy to train to train LLM LoRas. The reason you don't see LoRas on their own is because people who fine tune models merge them back into the base and ship the full model weights instead

Also see LoRAX, it's an inference engine that exactly does this, it can serve many LoRas off a single base model

▲

ffsm8 4 hours ago | parent [-]

I see, so you're off the opinion anthropic could provide a loras adaptation per user ready for use to enable timely response for every subscriber.

It is an interesting thought to explore. I admit I'm not a specialist whatsoever on the topic, from my perspective that sounds like an incredible challenging ops problem.... But again, I don't work for anthropic and maybe they're up to the challenge as you seem to think.

	▲	e12e 3 hours ago \| parent [-]
		If users get a fixed set/numbers of Paramus for LoRa - say 1-5% - it doesn't sound that crazy. The crazy part is probably loading, running a model pr users - but if the untrained (unspecialized?) model can somehow stay resident - might be possible to load adjusted weights on demand? I suppose each run now needs a different context pr user/customer anyway, for cache hit rate, if nothing else?

▲

a day ago | parent | prev | next [-]

[deleted]

▲

nullsanity 15 hours ago | parent | prev [-]

[dead]