> they're not going to re-scrape the content they already have

That's true for static content, but much of it is forums and other places like that where the main value is that new content is constantly generated - but needs to be re-scraped.

If only sites agreed on putting a machine readable URL somewhere that lists all items by date. Like a site summary or a syndication stream. And maybe like a "map" of a static site. It would be so easy to share their updates with other interested systems.