The issue is there's far more pages than could reasonably be stored: cgit is an old-school server-side renderer and there's a combinatorial explosion of possible pages it could render. The article calculates there's about a quadrillion different pages that could be 'crawled'