No. Open is open. Beyond DDoS protections, there should be no limits.

If load on the server is a concern, make the whole database available as a torrent. People who run scrapers tend to prefer that anyway.

This isn't someone's hobby project run from a $5 VPS - they can afford to serve 10k qps of readonly data if needed, and it would cost far less than the salary of 1 staff member.

> Open is open.

I’d then ask OpenAI to be open too since open is open.

Rate limiting is a DDoS protection.

You're talking about a tragedy of the commons situation. There is an organic query rate of this based on the amount of public interest. Then there is the inorganic vacuuming of the entire dataset by someone who wants to exploit public services for private profit. There is zero reason why the public should socialize the cost of serving the excess capacity caused by private parties looking to profit from the public data.

I could have my mind changed if the public policy is that any public data ingested into an AI system makes that AI system permanently free to use at any degree of load. If a company thinks that they should be able to put any load they want on public services for free, they should be willing to provide public services at any load for free.

The world is not black and white.