This only works well if you are creating sub agents with clean contexts for each task. If you constantly are switching models part way through some work then the whole session needs to be replayed each switch. You lose all the benefits of the context cache.

▲

xyzsparetimexyz 2 days ago | parent [-]

Well yeah, it'd require the models to be loaded on the same system and the cache to be shared between them somehow

▲

inigyou 2 days ago | parent [-]

Cache sharing is not possible. The numbers in the cache are completely specific to the model.

▲

xyzsparetimexyz 2 days ago | parent [-]

Currently. I'm sure that you could make a system where the cache values are a superset C of e.g. models A and B where C is probably bigger than max(A,B) but smaller than A+B

▲

wonnage a day ago | parent | next [-]

Reddit post so take with a grain of salt but this does seem possible. But unclear whether this is actually a viable architecture https://www.reddit.com/r/LocalLLaMA/comments/1t8s83r/nvidia_...

▲

inigyou 2 days ago | parent | prev [-]

It's equal to A+B. There is literally no sharing possible.

	▲	xyzsparetimexyz a day ago \| parent [-]
		Even when training both models together in a novel way?