But, under the assumption that the problem are indeed AI crawlers, of the things you listed only the search would be under increased load.

I am sure their pages are not entirely static either. The APIs are used by researchers and AI companies too. Also with search you end up having people trying to use it for RAG with AI. I have dealt with this all and there is no one dead simple solution to deal with things. AI crawlers are one part of things, but they also have increasing submissions, have to deal with AI generated spam papers, and all sorts of stuff. There's always this feeling here on HN that oh it is dead simple you just do "X" as if the people that are dealing with it don't know that.