Remix.run Logo
▲ atomicfiredoll 12 hours ago

iirc when I was dealing with reviewing traffic and AI scraping bots recently, I noticed something about Meta's: When a user links to a page, the bots that grab oembed data to generate the link preview are using the same User Agent string as the Meta AI crawlers that scrape content.

For context, oembed data about your page could be seen as information you're basically happy to share, so other sites can create things like previews.

That's not necessary, there are other companies that just use different UA strings for the different tasks. They appear to deliberately make it so that if you have nothing to do with Meta and block their content scrapers, you also break basic functionality for people who happen to share a link to your site in their products.