▲ | snowwrestler 2 days ago | |
Yes, it’s true. Most sites don’t have a forever cache TTL so a crawler that hits every page on a database-backed site is going to hit mostly uncached pages (and therefore the DB). I also have a faceted search that some stupid crawler has spent the last month iterating through. Also mostly uncached URLs. | ||
▲ | n3storm 2 days ago | parent [-] | |
Yeah, or an event plugin where spiders walks every day of several years... |