▲ | mrkeen 2 hours ago | |
Yep. Built a crawler, an indexer/queryprocessor, and an engine responsible for merging/compacting indexes. Crawling was tricky. Something like stackoverflow will stop returning pages when it detects that you're crawling, much sooner than you'd expect. |