They should provide torrents maybe? Rate limit access (also for seed), but that way the crawlers would also be incentivized to seed while they are finishing their fetch.

Monthly snapshots of the complete library maybe, and monthly diffs could work.