I think in general, "Writing on the Internet with the intent to make money" is effectively dead, or at least soon to be dead. AI+bots mean we now have the "infinite typewriter monkeys" from the thought experiment. With infinite supply, the price goes to zero.

We need to stop this treadmill of trying to "build reputation" and stop focusing on "symbolic capital" and "clout" and whatever else bloggers are going after. You're not going to get it, and even if you do, you're not going to be able to "monetize" it.

If you have a need to write, write. Maybe a handful of actual people will read it, maybe not. But, I wouldn't try to do it for a living. The reward will have to be the cathartic process of writing itself, and not in how much attention it gets, how much it "blows up" or how viral it gets.

>How is a newcomer supposed to build reputation now

Dead internet manifest.

I feel that pressure of not knowing how to definitively compete on the internet, especially when there's so much AI created noise.

I'm a copywriter and I used to get hired to write posts on behalf of founders on LinkedIn or for their company blog.

Now, the last three jobs I had were all focused on sending cold email.

By syndicating everywhere you can and inviting discussions (Reddit, HM, etc). The reality is that your blog post would've rarely been found anyway due to the SEO blog spam.

I managed to get some great discussions through sharing my posts on Reddit and HN.

I dunno. SEO spam made by human-robots already killed the first half of the internet in the late 2010s. Mostly driven by low-paid freelancers writing without expertise and plagiarizing other articles like crazy.

It has been virtually impossible to find real information in some areas if you don't personally know which website is reliable or not. That's why we devs used to go to StackOverflow, and people use site:reddit.com when searching Google. LLMs just exacerbated all of that, but it was already happening.

What's worse, is they train on your content, and very often you don't even get an attribution link. So the end user never even knows it was your site that provided the information and you never even get a single clickthrough. It's not like the SERPs where someone would click through, read your site, hopefully find it interesting and useful and come back.

It's going to be a serious problem and I've already seen sites that are down 90% in traffic simply because AI is scraiping them, answering the questions themselves and never providing a linkback.

[flagged]

The graphic in the article seems to be the only significant content.

Based on that I think it's more about requests from bots/scrapers having the greatest chance possible of hitting a cache before hitting the blog's origin/real host. Bots will hit some layer of Cloudflare first then they'll hit Fastly and then if not in Fastly they'll hit the Ghost blog's server.

To me, this makes a lot of sense if it's self-hosted but I also thought it was already the standard to shove your self-hosted blog behind a reverse-proxy and cache as much as possible.

And I'm not a professional web developer but all the extra caching layers for a static personal blog seem a bit overkill.

Aside from the graphic, the article is a lot of words about engaging with an LLM to get a full understanding of how caching works for their blog hosting and how it enabled them to change their setup for the better.

It's kind of hard to understand because there are no words about what they actually did or how what they actually did was better.

From the post:

> If you care about how your content moves through the world now, including through AI systems, you have to care about caching. Not as a performance optimisation for human browsers, but as infrastructure for machine readership.

Page load time can impact index coverage (depth of crawl), freshness (revisit rate), and ranking.

I am on a low tier Ghost subscription, I could not rewrite some of the HTML. So I do this with Cloudflare and then cache it again.

Yes, the architecture setup is generated by ChatGPT but in itself it says what it needs.

Ghost blogs platform already has a cdn as part of their product so, makes even less sense to me.

Maybe we're being trolled and the op is crowd sourcing solutions by posting something ridiculous and getting us to put solutions up

This is why I don't use those serverless setup.

$4 hetzner vps can serve tons of request if you put cloudflare in front of it

I host my own runners for CI and artifcat building on Hetzner VPS (spun on demand).

People are easily lured by pay as you go plans on serverless and other cheap to get started managed services and end up racking huge bills.

This is same reason I don't use stack driver or cloud monitoring and prefer to use graphana + loki + Prometheus setup

My setup cannot be misconfigured and end up racking huge bills.

I also block all AI crawlers. I am not sure why I should give them my content for them to rip it off and make money from it through training or agents. Sadly, a lot of AI companies are trying to make requests indistinguishable from regular browsers from residential connections, so unfortunately I have to use Cloudflare to block them.

Ideally I'd make the content available to crawlers for training open models, but that seems to be nearly impossible. It would be possible if other AI companies behaved.

This works for a few weeks to months. Then they detect your site is hostile to them and enable evasion mode, with random IP addresses and user-agent strings. Proxies are expensive so at least they're losing money.

*some AI crawlers. Not many

I started blocking some of them. But for now I want to improve visibility before further blocking or optimising. The dashboard helps with this.

It is good to make a proper distinction, in the ChatGPT context, between crawlers and agents. The crawlers go for the content to build a new model, the agents serve content to users. The last one can be very useful.

TLS fingerprinting may identify what they actually are

Same ratio roughly. 80% Crawlers and agents, 20% human. Loads of the agents actually serve the content to humans, mostly in ChatGPT.

Thanks. It seems to be very local/incidental. The page works from the locations I can test, but I’ll check whether one edge cache or request path served a bad response.

Same here via VPN. No VPN, and I get the actual content.

Caching gone wrong.. (Works for me)

I recently un-denylisted AI crawler UAs from my blog's robots.txt, for several reasons:

- My blog is static content and it costs me ~nothing to serve the requests.

- The bots were ignoring robots.txt anyway.

- If there's ultimately a human driving the bot (e.g. someone asking "summarise this article"), I don't mind.

- It's like trying to block search engines. Just as I want my blog to turn up in search results, I want agents etc. to know it exists, too.

My original motivation for denylisting, years ago, was that LLMs were simply not very good, so training-set scrapers seemed like all downside with no upside.

Why would bots and the people operating them care what you want?

Like it or not, but it's the intermediary layer now, between you and your users.