On this note - what's "i.i.d."?

I believe this is referring to the probability concept of "Independent and Identically distributed".

In the usage in the book/page, it seems to refer to how tasks/problems are run in parallel and the learning averaged, whereas the author is advocating these problems run sequentially.

https://en.wikipedia.org/wiki/Independent_and_identically_di...