Fun experiment:

Have an LLM “guess” random URLs seeded with words from a dictionary, iterating over each word and guessing a URL.

It guesses a lot of correct URLs. This is one method of “URL hunting” that doesn’t involve a 3rd party list or index.

Then just scan those pages for other URLs, visit them, and add a tally every time you come across a URL (for page rank).

Then search anything, see what the results are. You have invented a dark web search engine.

> dark web

Guessing random words and .onion at the end probably won't get you anywhere, although onion is also quite a random word...

Deep Web https://en.wikipedia.org/wiki/Deep_web

  The deep web,[1] also known as the invisible web[2] or the hidden web,[3] is the parts of the World Wide Web whose contents are not indexed by standard web search-engine programs.[4] This is in contrast to the surface web, which is accessible to anyone using the Internet.[5] Computer scientist Michael K. Bergman is credited with inventing the term in 2001 as a search-indexing term.[6]