The needed inovations for any potential new business model of the internet are not technical.

Even if some system of making crawlers pay for a content access, how could you be sure it's never used without your permission? It's not like DRM solved internet piracy.

And even if a technical solution were possible, it's natural to see this level of cynicism of how an internet giant would handle it.