Remix.run Logo
7moritz7 19 hours ago

The scale. How many tools do you know that can query the content of all arxiv papers.

eamag 12 hours ago | parent [-]

Doesn't look like the scale is there, even for HN:

> Currently have embedded: posts: 1.4M / 4.6M comments: 15.6M / 38M That's with Voyage-3.5-lite

Xyra 3 hours ago | parent [-]

The scale is there. I'm scraping, cleaning, token efficientizing dozens of sources every single hour. The lack of monies for embedding everything was a temporary problem.