Remix.run Logo
dpoloncsak 2 days ago

I'm not in support of the practice laid out in the article, but we're talking about robots.txt, right?

I guess it was written for a less technical audience, but it makes it seem like they have JS or 'code' was specifically written to hide these from web crawlers.

It makes more sense, in context, that companies could be unaware. Sure, a 'noindex' doesn't just show up, but how many were old configs disallowing *, and only allowing indexing on a few sites

Edit: I didn't see the screenshot section. Most (of the few I spot-checked) are, in fact, noindex. I stand corrected