Everyone seems to be dismissing this without reading it. As a content publisher, I am very interested in any proposal that results in me getting residual payments from AI scrapers. That would indeed be a new internet business model.

This is also being attempted by RSL with their “Crawler Authentication Protocol” (https://rslstandard.org/guide/web-crawlers) for demanding proof of licensing from scrapers and RSL Collective (https://rslcollective.org) for providing the licensing itself. The missing piece there is the ability to detect scrapers with high accuracy without punishing regular browsing humans.

As an engineer who works at a medium-large publishing company, am also interested in this direction.

I have fully read Cloudflare's proposal, as I have RSL, and L402.

In its current state, RSL is not a full solution, but can be. The spec does not cover payments. Its basically just a new token type, which crawlers would register for to crawl your content, and predefined responses for more detailed states. But nothing around payments, yet.

Cloudflare's solution is basically the same as L402. This is very sad for me, because their attempt is basically one of locking down the ecosystem in order to create a monopoly.

If they were really about creating a fair internet where "pay per crawl" is possible (it already was with L402), they would openly support L402, implement it for their gateway, and try to push the industry towards it. But no, they just want the biggest piece of the pie; a pie that doesn't even exist yet.

> without punishing regular browsing humans.

As a content consumer, I'm also hoping to be part of the ecosystem. I already use Patreon a lot as "AdBlock absolution", but it doesn't fix the market dynamics. Major content platforms tend to stagnate or worsen over time, because they prefer to sell impressions to advertisers than a good product to consumers.

I'm really excited by this post. I love the vision. especially the vision of using what is already known about the gaps and what is not known in having content created to fill those gaps. There are definitely some gaps in LLM knowledge.

> As a content publisher

It always bothers me a little when people describe themselves as a content publisher or content creator. Why content? Why not say you are a short film maker or writer or musician or videographer or journalist or painter or photographer or ...?

In the past, your work has been merely content to the people who are looking for bits -- any bits -- to place between and around the stuff they really care about: ads. These days I suppose your work may be seen by some as grist for the AI mill.

By calling your work content you are acceding to the idea that your work is only valuable in other contexts. It isn't! I bet it's something you made with care and thought.

Or maybe I misread and you really are a content publisher. You're the person running the CMS who is looking for other people's work to monetize. If so, nevermind.

I think as a publisher, you should be worried that Cloudflare is taking the first step towards the 2025 version of what “rights holders” have done to the movie and entertainment industry.

Imagine trying to create and distribute a movie without the backing of your local distributor/broadcasting cartel. Only instead of gating the movie theatre, Cloudflare is the single access to all things web. Also, based in the US, so good luck producing content their government doesn’t like.

I like the IDEA of residual payments for content on paper. Brave browser tried this, but it didn't get enough traction from publishers of note to be viable. But I think, at least in theory, the idea of me as a consumer having some money attached to my eyeballs for the person/people who created the content I'm seeking out is a good one. Paywalls are annoying and awful, and unfortunately they seem to encourage data hording and selling practices that are unsavory. But I like the model of simply opening the browser, visiting a page, and the creators getting some royalty from that. The system we have now is too exploitative - it extracts more value that it creates.

I mean, you shouldn't need to read a wall of text to understand what a business model is.

It should be front and center, communicated clearly, and easy to understand. If I was an investor, the lack of clarity after reading a few paragraphs would concern me.

> As a content publisher, I am very interested in any proposal that results in me getting residual payments from AI scrapers.

So you use a public resource and presumably like the upside that comes with sharing on it, but you want to limit uses now that someone has found one that you don't like, so you're fine with degrading that public resource?

You could always share things privately or behind a paywall if you don't want them available publicly. But people seem to want to have their cake and eat it too.

I get why a hosting provider would want to limit crawlers to save bandwidth. The "creator" angle is just greed.