Remix.run Logo
alenmangattu 4 days ago

I’ve spent the last 3 months building a crawler to index the public parts of Telegram (https://telehunt.org). The native search is essentially a black box that favors the top 0.1% of bot almost invisible. The Tech: I had to deal with rate limits and the lack of a global 'sitemap'. I’m currently using a hybrid approach of metadata scraping to keep the index fresh. The Goal: It’s an experiment in making 'un-indexable' bot data discoverable.

duskwuff 7 hours ago | parent | next [-]

You may be overestimating the number of bots that meaningfully exist. The vast majority of bots (and public channels) on the platform are nonfunctional and/or spam.

Antibabelic 9 hours ago | parent | prev [-]

Where is the search engine? The site says that it's a bot directory.

renegat0x0 7 hours ago | parent [-]

wikipedia "A search engine is a software system that provides hyperlinks to web pages, and other relevant information on the Web in response to a user's query".

I think there can be different expectation connected to this term. It seems to be a "search engine" for bots. Bot directory does not have to have "search" functionality, right?