>>Opus downloaded all comments of HackerNews

I'm actually surprised this was that easy to do, ngl

HN's Firebase-based API (https://github.com/hackernews/api) makes that easy, until you want to reconstruct the threads (i.e. who replied to who) and then it's annoying.

For years* I've wanted to replace that API with one we operate ourselves, which will simply return a JSON version of any HN URL.

I've finally started working on this! Perhaps my favorite thing about LLMs is how they make projects possible which would otherwise not have gotten started.

Edit: perhaps I should explain what I mean by that last bit.

The way I want this new API to work is (1) HN should use it internally to generate HTML pages, and (2) there can't be any regressions - nothing should change in the web version. This is a pain to get right, as it's easy subtly alter the details, and it's annoying to regression-test everything. Also, (3) with a design like this, it is easy—and critical not to—leak data that ought to remain private (e.g. people's IP addresses)

This is the sort of pain and annoyance that LLMs are amazing at taking care of. I get to focus on the parts I want to think about—the design and (yes) the code—and have a tireless assistant do the gruntwork and review. Without this I don't think I'd have gotten around to working on it.

* (edit: over a decade in fact - see collapsed reply)

There turns out to be an export on HuggingFace as well as a bunch of parquets.