I needed to thoroughly test rerankers on my companies rather unique corpus.

Opus and I wrote a parallelized test harness and labeled groundtruth in around 2 hours.

In 2022 that would've likely been all I did for a couple sprints

I encounter this regularly and it still feels weird.

That sense that you did something better in a few days than you would have in a month 5 years ago. It's like buying a table saw for wood working.

One crazy thing I think about often is how there are so many correctness and testing harnesses that would have taken weeks to build in the past so we simply never would have. We'd just do our best then wait and see what comes to the surface. This is a huge part of what makes it possible to actually make better software with LLMs in my opinion. It isn't just 'LLM codes better than I ever could' (that's often untrue still) but 'LLM enables me to make assertions about the program to degrees that would have been absurdly impractical in the past'. It's huge

Yes 100%. This morning I casually prompted Codex to drive the browser to complete extensive performance testing in-situ that would have literally been weeks of work before. Probably in reality it just wouldn't have been done, and performance guarantees would have been attempted up front via more careful design.

In this case the design was also AI generated, and there were limited wins to be found because the design was already superb.