| ▲ | hombre_fatal 2 days ago | |
My forum traffic went up 10x due to bots a few months ago. Never seen anything like it. > Loading static pages from CDN to scrape training data takes such minimal amounts of resources that it's never going to be a significant part of my costs. Are there cases where this isn't true? Why did you bring up static pages served by a CDN, the absolute best case scenario, as your reference for how crawler spam might affect server performance? | ||
| ▲ | senko 2 days ago | parent | next [-] | |
Not OP, but many technologies nowadays push users to use a server-side component when not needed. An example is NextJS where you're strongly encouraged[0] to run a server (or use a platform like Vercel), even if what you're doing is a fairly simple static site. Combine inconsiderate crawler (AI or otherwise) with a server-side logic that doesn't really need to be there and you have a recipe for a crash, a big hosting bill, or both. [0] People see https://nextjs.org/docs/app/guides/static-exports#unsupporte... and go "ah shucks I better have a server component then" | ||
| ▲ | dehrmann 2 days ago | parent | prev [-] | |
> My forum traffic... > Why did you bring up static pages served by a CDN... This is easier said than done, but pushing the latest topic snapshot to the CDN whenever a post is made is doable. | ||