| ▲ | One Wikipedia page costs your AI agent 68,000 tokens | |||||||
| 12 points by arhamislam5766 13 hours ago | 6 comments | ||||||||
i use claude code daily and measured what pages cost it while doing research. an average wikipedia article, for instance, is 68,240 tokens of raw html (tiktoken); nike's homepage is 353,000. claude code's built-in webfetch handles the easy case well. it summarizes wikipedia to about 950 tokens and clears cloudflare on some sites like indeed and ticketmaster. but, and there's always a but, on js-rendered and some anti-bot pages it returns nothing. quotes.toscrape.com/js gives "no quotes found"; nike.com gives a 403. your agent then dumps the raw html back into context and still fails. (note: i have also had cases where i read through the chat at the end and saw that it failed and just pulled from either training data or stale caches from other sources) so i worked on building an open-source stealth browser (recompiled chromium) that runs as an mcp for claude code, cursor, and claude desktop. essentially all i have to change form my end is add the mcp, and it returns the cleaned up tokens while also beating detection: the js quotes come back in 285 tokens, nike in about 700 instead of a 403. there is still stuff i am actively working on: there's no residential egress yet, and it won't beat kasada-style walls. it's for agents, qa, and research. repo and the reproducible benchmark: https://github.com/tiliondev/fortress/tree/main/mcp i'm the author and here for feedback. | ||||||||
| ▲ | chonghaoju 10 hours ago | parent | next [-] | |||||||
stripping to markdown with Jina Reader or Trafilatura before passing to the agent cuts that 68k down to ~3-5k for most Wikipedia pages, and handles the JS-rendered case too. | ||||||||
| ||||||||
| ▲ | arthurcolle 12 hours ago | parent | prev | next [-] | |||||||
one of the early jina.ai products was/is reader api, and they trained ReaderLM for this purpose. definitely a good idea to check out existing implementations | ||||||||
| ▲ | m463 11 hours ago | parent | prev | next [-] | |||||||
> on js-rendered and some anti-bot pages it returns nothing yeah, and my (human piloted) browser gets blocked by so many websites now. I routinely get "Sorry." when trying to log into hn. sigh. | ||||||||
| ||||||||
| ▲ | mkbkn 8 hours ago | parent | prev [-] | |||||||
And how much water? | ||||||||