Put a CDN in front and let them absorb the load? Seriously, this is static content, which is so cheap to serve it should be free.

▲

ninglor 4 hours ago | parent | next [-]

This is not meaningfully static content. Look at the charts in TFA. There is a combinatorial explosion of distinct URLs which the crawlers can and do request.

▲

jopsen 4 hours ago | parent | prev | next [-]

They allow you to diff commits, which is an awesome feature.

But if bots a crawling diffs between all possible commits it's crazy.

CDN will do nothing, because it's new urls each time. You can maybe find a CDN provider that block bots.

	▲	inigyou an hour ago \| parent [-]
		No CDN blocks bots. Some claim to. None do.

▲

grep_it 4 hours ago | parent | prev [-]

Did you read the article?

“[…]because we can generate 1.2 METRIC BAJILLION valid URLs just for a single fork of linux.git.”

▲

jwilk 2 hours ago | parent [-]

From the HN guidelines <https://news.ycombinator.com/newsguidelines.html>:

> Please don't comment on whether someone read an article. "Did you even read the article? It mentions that" can be shortened to "The article mentions that".

	▲	inigyou an hour ago \| parent [-]
		What if it's really obvious they didn't read the article?