My understanding is that in addition to your comment and the development of a method to separate the training data for distributed learning, the latency/bandwidth of systems connected on the internet is a challenge, too. Information has to be sent around before and after any hypothetical number crunching.

You would probably not be able to go down to the scale of a single PC, but it should be possible to train models focusing on different specialties on different nodes and then have them periodically "mix" together.