| ▲ | michaelmior 4 hours ago | |||||||||||||
> I'm surprised that Cloudflare hasn't started hosting a pre-scraped version of websites that use Cloudflare's proxy It's entirely possible that they're doing this under the hood for cases where they can clearly identify the content they have cached is public. | ||||||||||||||
| ▲ | janalsncm 3 hours ago | parent | next [-] | |||||||||||||
How would they know the content hasn’t changed without hitting the website? | ||||||||||||||
| ||||||||||||||
| ▲ | binarymax 4 hours ago | parent | prev [-] | |||||||||||||
Based on the post, it seems likely that they'd just delay per the robots.txt policy no matter what, and do a full browser render of the cached page to get the content. Probably overkill for lots and lots of sites. An HTML fetch + readability is really cheap. | ||||||||||||||