There are many of these stream-experts-from-SDDD projects now. What's needed is a guide to all of them and maybe for someone to take the best features of each of them and merge them into one uber project.

My impression based on a cursory look at the research is that the caching algorithm is absolutely vital, non-obvious, model-dependent, and potentially prompt-dependent to make sure you're holding onto the right experts.

It doesn't look like a solved problem yet.