I'd love to see people figuring out how to build models from several smaller ones. We could then train small specialized models and deploy setups more optimized for any given task. Modular LLMs should be a thing.

This is something I've been trying to figure out for a bit, some models are really good at instructions, but their context window is too small, I do wonder if having a cluster of smaller models would be feasible. Been building a custom coding harness so once its nice and polished I might experiment with this more.