Remix.run Logo
▲ xnorswap 3 hours ago

I agree completely, there was a much better talk on this by Lauren Tan: https://www.youtube.com/watch?v=O_zcc6Xl12g

A key pillar was introducing things like static analysis to provide cheap deterministic checks as the first goal and only then moving to other forms of verification.

▲fg137 an hour ago | parent | next [-]

The talk makes that sound something new, but every team I know has already adopted that before AI. I really don't get what value that talk provides -- I learned nothing after watching it.

▲xnorswap an hour ago | parent [-]

You are fortunate to have always worked in highly effective teams and organisations.

This reminds me of when "The Joel Test" was first published and it was met by people unbelieving there was a world out there who weren't even using source control at the time, let alone a bug-tracker.

▲simianwords 3 hours ago | parent | prev | next [-]

You are not agreeing with the OP. You are just saying that there are many levels to testing and each give their own swiss cheese level of verification.

Sure you can have linting, compilation, unit tests, e2e tests. Do you really think Anthropic doesn't do this?

▲xnorswap 2 hours ago | parent | next [-]

I was agreeing that this page as presented felt like it was from the POV of people will unlimited tokens to burn.

In particular constructing ( and often deconstructing on change ) differential tests is very expensive relative to cheaper things to set up.

I felt this article's argument was poorly presented because of this seeming contradiction. The video I linked was much clearer about what works and what doesn't when scaling to inhuman levels of PR merges.

▲simianwords 2 hours ago | parent [-]

I don't agree with "very expensive" claim. My friend works in a f500. They do releases like 2-3 times a week. Assuming each release uses $500 of tokens (higher estimate) just for verification, that's like $6000 a month and that's just a small percent over the tokens used for coding. What's the big deal? Their team brings in way more money than that - its a drop in the ocean.

I also know of companies that run very expensive test suites using spot instances in AWS that cost around $100 each time. No one complains about it.

I think all the posts complaining about expenses forget to compare the orders of magnitudes.

▲xnorswap 2 hours ago | parent | next [-]

It's not the per-run costs I worry about, it's the costs of churn.

As soon as Oracle Tests fail, the LLM has three options:

1. Rewrite the test - This undoes the benefit of the test in the first place 2. Implement workarounds and quirks to match the oracle, often at the expense of recreating buggy behaviour 3. Implement both corrected behaviour and quirks-mode behaviour to meet the oracle, raise further jobs to formalise the switch over and change of behaviour.

( And this "change of behaviour" might be a few pixels out of place! Or it could be a much more significant change. When trying to meet a goal, LLMs will often treat these with the same zeal. )

If you're working interactively, you can catch it and steer it to what's best for the feature. If you're fully automating development, then you're at the mercy of whatever the model does at that moment.

Oracle-based testing has its place, but I don't think it's good for large scale automated testing.

Also, you're talking about $500/release 2-3 times a week. That's fine, that's less than the cost of an employee.

Anthropic and others are talking about how to scale to merge 2000-3000 PRs per week.

At that scale, that cost per merge is $6M/month, surely a cost you'd want to minimise if you can cut them by half with better type-checking, static analysis, and other cheap deterministic tests, even if that's still a small cost of the whole project.

If nothing else, it's pure waste. Our planet is (mostly metaphorically) burning, and I believe that more efficient IT processes and development practices would go a long way to helping mitigate some of that.

▲cicko an hour ago | parent | prev [-]

Come back to Earth, please. Not everyone wants to suddenly spend $6K more per month on deployments.

▲sceptic123 2 hours ago | parent | prev [-]

If you take the post at face value they are saying some of those tests aren't valuable

▲rolosa 2 hours ago | parent | prev [-]

That developer brags about slopping out 2000+ PRs a month, how is anything they have to say useful?

▲xnorswap an hour ago | parent [-]

Slopping out 2k+ PRs/mo while also claiming to reduce defect rates.

That seems like something to pay attention to, even if you prefer to treat it as a cautionary tale with your popcorn at the ready for when you expect it all comes crashing down.

▲rolosa 41 minutes ago | parent [-]

At 2000 PR/mo, assuming a "standard" 8 hour per day, 5 days a week workweek with no time factored for lunch, meetings, breaks, or PTO... literally all they do for 8 hours a day, 5 days a week, is submitting PRs. That works out to roughly 5 minutes to slop, review, and document a single PR, approx 92 PRs per work day. That tells me they aren't doing shit.

▲xnorswap 21 minutes ago | parent [-]

I suspect that an ever increasing number of tickets being raised are being raised by agents themselves, nitpicking and being unable to deal with the idea of implementations having trade-offs.

It doesn't take 5 minutes to prompt claude, "Please pick up Ticket #1234567".

With modern models, you rarely need much more than that. Perhaps checking in an hour later with a quick prompt, "Progress report?" if it's still working on it.

Claude having written the ticket in the first place, defined acceptance criteria. Perhaps a human has skimmed it for a go/no-go decision, perhaps not.

The point is that at that scale you can't humanly read the code, so you need to define verification strategies that work for the machine.

Precisely so that you're not just running on pure vibes.