However, so-called AI crawlers are not the same as crawlers of yore. They hit live pages every time a user prompt triggers a web search.

This Web Search API, unlike an AI crawler, only fetches periodically. It feels like a step in the right direction for managing resource strain across the internet. If only the LLM giants could do something similar.

The "LLM giants" do both. They have to create an index to know which pages to even fetch in the first place. A simple web search does not actually necessarily hit any pages it's only when it does a page fetch based on that search