See the recent development of DiLoCo at Nous Research and Prime Intellect.

oh that's very cool, exactly the kind of stuff we need