| ▲ | OtherShrezzing 3 hours ago |
| This article feels like the result of having unlimited tokens at your disposal. It’s got some very expensive practices, where near-free practices would produce the same or better results. There’s a long section in the middle where the article describes per-change verification via skill files and “ask Claude to use the chrome-for-Claude tool” when checking if changes have introduced regressions. But there’s already tools which do this without spending tokens tokens. Playwright will achieve equal quality but deterministic results across the same verification attempts. It is write-once-use-often, so has a marginal cost near zero. |
|
| ▲ | xnorswap 3 hours ago | parent | next [-] |
| I agree completely, there was a much better talk on this by Lauren Tan: https://www.youtube.com/watch?v=O_zcc6Xl12g A key pillar was introducing things like static analysis to provide cheap deterministic checks as the first goal and only then moving to other forms of verification. |
| |
| ▲ | fg137 an hour ago | parent | next [-] | | The talk makes that sound something new, but every team I know has already adopted that before AI. I really don't get what value that talk provides -- I learned nothing after watching it. | | |
| ▲ | xnorswap an hour ago | parent [-] | | You are fortunate to have always worked in highly effective teams and organisations. This reminds me of when "The Joel Test" was first published and it was met by people unbelieving there was a world out there who weren't even using source control at the time, let alone a bug-tracker. |
| |
| ▲ | simianwords 3 hours ago | parent | prev | next [-] | | You are not agreeing with the OP. You are just saying that there are many levels to testing and each give their own swiss cheese level of verification. Sure you can have linting, compilation, unit tests, e2e tests. Do you really think Anthropic doesn't do this? | | |
| ▲ | xnorswap 2 hours ago | parent | next [-] | | I was agreeing that this page as presented felt like it was from the POV of people will unlimited tokens to burn. In particular constructing ( and often deconstructing on change ) differential tests is very expensive relative to cheaper things to set up. I felt this article's argument was poorly presented because of this seeming contradiction. The video I linked was much clearer about what works and what doesn't when scaling to inhuman levels of PR merges. | | |
| ▲ | simianwords 2 hours ago | parent [-] | | I don't agree with "very expensive" claim. My friend works in a f500. They do releases like 2-3 times a week. Assuming each release uses $500 of tokens (higher estimate) just for verification, that's like $6000 a month and that's just a small percent over the tokens used for coding. What's the big deal? Their team brings in way more money than that - its a drop in the ocean. I also know of companies that run very expensive test suites using spot instances in AWS that cost around $100 each time. No one complains about it. I think all the posts complaining about expenses forget to compare the orders of magnitudes. | | |
| ▲ | xnorswap 2 hours ago | parent | next [-] | | It's not the per-run costs I worry about, it's the costs of churn. As soon as Oracle Tests fail, the LLM has three options: 1. Rewrite the test - This undoes the benefit of the test in the first place
2. Implement workarounds and quirks to match the oracle, often at the expense of recreating buggy behaviour
3. Implement both corrected behaviour and quirks-mode behaviour to meet the oracle, raise further jobs to formalise the switch over and change of behaviour. ( And this "change of behaviour" might be a few pixels out of place! Or it could be a much more significant change. When trying to meet a goal, LLMs will often treat these with the same zeal. ) If you're working interactively, you can catch it and steer it to what's best for the feature. If you're fully automating development, then you're at the mercy of whatever the model does at that moment. Oracle-based testing has its place, but I don't think it's good for large scale automated testing. Also, you're talking about $500/release 2-3 times a week. That's fine, that's less than the cost of an employee. Anthropic and others are talking about how to scale to merge 2000-3000 PRs per week. At that scale, that cost per merge is $6M/month, surely a cost you'd want to minimise if you can cut them by half with better type-checking, static analysis, and other cheap deterministic tests, even if that's still a small cost of the whole project. If nothing else, it's pure waste. Our planet is (mostly metaphorically) burning, and I believe that more efficient IT processes and development practices would go a long way to helping mitigate some of that. | |
| ▲ | cicko an hour ago | parent | prev [-] | | Come back to Earth, please. Not everyone wants to suddenly spend $6K more per month on deployments. |
|
| |
| ▲ | sceptic123 2 hours ago | parent | prev [-] | | If you take the post at face value they are saying some of those tests aren't valuable |
| |
| ▲ | rolosa 2 hours ago | parent | prev [-] | | That developer brags about slopping out 2000+ PRs a month, how is anything they have to say useful? | | |
| ▲ | xnorswap an hour ago | parent [-] | | Slopping out 2k+ PRs/mo while also claiming to reduce defect rates. That seems like something to pay attention to, even if you prefer to treat it as a cautionary tale with your popcorn at the ready for when you expect it all comes crashing down. | | |
| ▲ | rolosa 42 minutes ago | parent [-] | | At 2000 PR/mo, assuming a "standard" 8 hour per day, 5 days a week workweek with no time factored for lunch, meetings, breaks, or PTO... literally all they do for 8 hours a day, 5 days a week, is submitting PRs. That works out to roughly 5 minutes to slop, review, and document a single PR, approx 92 PRs per work day. That tells me they aren't doing shit. | | |
| ▲ | xnorswap 22 minutes ago | parent [-] | | I suspect that an ever increasing number of tickets being raised are being raised by agents themselves, nitpicking and being unable to deal with the idea of implementations having trade-offs. It doesn't take 5 minutes to prompt claude, "Please pick up Ticket #1234567". With modern models, you rarely need much more than that. Perhaps checking in an hour later with a quick prompt, "Progress report?" if it's still working on it. Claude having written the ticket in the first place, defined acceptance criteria. Perhaps a human has skimmed it for a go/no-go decision, perhaps not. The point is that at that scale you can't humanly read the code, so you need to define verification strategies that work for the machine. Precisely so that you're not just running on pure vibes. |
|
|
|
|
|
| ▲ | pu_pe 2 hours ago | parent | prev | next [-] |
| > It’s got some very expensive practices, where near-free practices would produce the same or better results. I think tokens are near-free already, even for those of us outside Anthropic. More importantly, your philosophy comes from a time where thinking had to be front-loaded: you try to predict everything that can go wrong, then design deterministic tests that can catch those things and a decision tree to figure out what to do in each case. This approach was the only viable way until now, but maybe that will change. Having the equivalent of a senior engineer thoroughly review each error with full context of what the application and feature is trying to achieve could perhaps uncover different insights, or cover edge cases more smoothly. |
| |
| ▲ | Grimburger an hour ago | parent | next [-] | | > tokens are near-free already How long do you think that will last? Am over 11 billion tokens this month on a subscription. In API usage its close to $5000 and there's still 13 days left in the month. Eventually the music in musical chairs stops. Already many anecdotal reports of the always opaque limits being lowered across the board. On a gemini pro account for personal use I can use a weeks worth of Argon in about 20 minutes on agy in the last month. ChatGPT Pro I can use the weekly limit in under 2 days, this is with significant context clearing and handoffs. The cost to provide frontier models isn't cheap and it's entirely subsidised by a winner-takes-all mentality from investors which in all likelihood won't come about. | | |
| ▲ | olkre 24 minutes ago | parent [-] | | The cost halves every year so far. Current level of llm will get cheaper | | |
| ▲ | curt15 15 minutes ago | parent [-] | | Cost or price? ANT's shareholders and creditors aren't going to wait forever for ROI. |
|
| |
| ▲ | skohan an hour ago | parent | prev | next [-] | | It's great to have that option available, but it's incredibly inefficient to just throw GPU hours at a problem that can be solved deterministically. | | |
| ▲ | pydry an hour ago | parent [-] | | Good for anthropic margins though. The absolute last thing they want with the upcoming IPO is people conserving tokens and ruining the already dying hockey stick narrative. |
| |
| ▲ | castis an hour ago | parent | prev | next [-] | | > Having the equivalent of a senior engineer thoroughly review each error with full context of what the application and feature is trying to achieve could perhaps uncover different insights, or cover edge cases more smoothly. No one is debating this. The debate is that an agent performing validation work via its own chrome plugin and internal reasoning will always cost more than performing that same validation work with deterministic tools. | | |
| ▲ | pu_pe an hour ago | parent | next [-] | | It isn't the same work, though, so the question is whether the added cost is worth it. Yes it costs more, but the price of tokens has fallen 100x in the past two years and it will continue to drop. | |
| ▲ | chrisweekly an hour ago | parent | prev [-] | | "non-deterministic" -> "deterministic" right? | | |
| |
| ▲ | jgalt212 2 hours ago | parent | prev | next [-] | | As far as timelines go both property based tests and fuzzers predate this token maxing strategy. | | |
| ▲ | bayindirh 2 hours ago | parent [-] | | But they are old, well tested, and not AI based, hence bland. In this age of AI, anything well understood and effective while being efficient is out. These tokens must be used, atmosphere must be heated. It's November. Winter is coming. |
| |
| ▲ | skydhash 2 hours ago | parent | prev [-] | | > you try to predict everything that can go wrong, then design deterministic tests that can catch those things and a decision tree to figure out what to do in each case No real software engineer think that way, as things that can go wrong are near infinite. Instead you design the right thing and write tests that verify that it’s indeed the right thing and add some more that are the wrong thing, especially at the edge of definition. Then you rely on the dev to not make very obvious errors like coding something that is not in the requirements or is completely opposite. |
|
|
| ▲ | urbsgpw an hour ago | parent | prev | next [-] |
| I got this feeling a couple of times this year with their sharing of best practices. They seemed theoretically interesting, but not if you're cost-conscious. It seemed like thinking with an unimited budget. And dont get me wrong, maybe that's the way to go since you assume costs of compute will fall down eventually (and these systems will become more optimized), but it would be completely untenable to run if you don't have a similar budget constraint (or lack thereof). |
|
| ▲ | arbll 3 hours ago | parent | prev | next [-] |
| What makes you think they are not using Playwright ? For me this article is more aimed at devs that are still stuck treating code generation and verification as two separate tasks. Giving agents the right tools (including playwright) to do both at once makes things more token efficient, not less. |
|
| ▲ | LelouBil 2 hours ago | parent | prev | next [-] |
| I had a developer friend who wanted to automate gym reservations at night, his first idea was to make use of ChatGPT Computer use, but he encountered refusal of the AI to enter his password, and it was difficult to convince it to do it without approval as well. I just told him to make the AI write a small selenium python script to do the clicking and it worked amazingly. |
| |
| ▲ | abirch 2 hours ago | parent | next [-] | | It seems like a new stack where you try to push things down as low as possible for efficiency. [ AI ]
[ Software ]
[ Kernel/OS]
[ Hardware ]
| |
| ▲ | saagarjha an hour ago | parent | prev [-] | | Fun fact: Codex computer use is trained to not enter passwords. When it was being designed this was considered to be a very cheap way to ensure the model would not go poking around places it shouldn’t be, instead of constantly trying to check whether the action being performed was authorized or not. People have spent a lot of time marking out their UIs for sensitive actions and it’s worth using it :) |
|
|
| ▲ | bitlad 3 hours ago | parent | prev | next [-] |
| They are living in the future. I think future AI cost will be as cheap as doing an API call in classic sense. |
| |
| ▲ | tpetry 2 hours ago | parent | next [-] | | Many API calls can be answered by something as underpowered as an Raspi. But AI needs very expensive GPUs to run. So no way it will be as cheap. That is just wishfull thinking. | |
| ▲ | ben_w 2 hours ago | parent | prev | next [-] | | Quite possibly. But, a thought I had some years back: In the future when AI is able to do everything, there will not be one single super-AI that does it all, but a whole ecosystem of different AI. This is the same as how in the biological world, while there are various degrees of generalists, there's no single cell, plant, or animal which does absolutely everything. If our body needs more blood, it makes more blood cells, it doesn't repurpose intestine or skin for the task. Similar argument in economics, comparative advantage: an AI which can do everything is best used for the things which only it and no other AI can do, and leaving to other (smaller, cheaper) AI relatively simple tasks like face recognition. Even where AI can do this, that doesn't mean the best solution is AI rather than a traditional API. | | |
| ▲ | brabel 2 hours ago | parent [-] | | You are dangerously close to dismissing the Bitter Lesson. https://en.wikipedia.org/wiki/Bitter_lesson | | |
| ▲ | ben_w 2 hours ago | parent [-] | | Hardly. First, last I checked AlphaZero is a much better chess player than any LLM, and by a large margin. Second, this isn't even about finding the best model for small tasks, it's about best use of finite resources, compute or energy, which means that it still holds even if a future AI (call it Alice) did everything better than AlphaZero does chess, it would still make sense for Alice to offload chess to a merely-good model like AlphaZero so that Alice could spend more time doing the things that nothing else besides Alice could do: https://en.wikipedia.org/wiki/Comparative_advantage |
|
| |
| ▲ | slowhorse 2 hours ago | parent | prev [-] | | So the future is about replacing deterministic procedures with stochastic procedures? On some level you are right, LLMs and the agents built with the various models and architectures will get cheaper. But there are some significant drawbacks by introducing this risk, regardless of cost. The future will need the capacity to understand this too. |
|
|
| ▲ | cyanydeez 3 hours ago | parent | prev | next [-] |
| Chinese gonna beat anthropic because they gluttonous use of AI. |
|
| ▲ | hypendev 2 hours ago | parent | prev | next [-] |
| Exactly. If you're doing this, the agents should create deterministic tests from its process rather than Agent tests or just writes deterministic tests for start. This is a bit of an absurd approach. The issue I find with these articles is that they are basically the same boat as "Facebook Engineering" articles were 20 years ago. People will be "oh because Anthropic does it it must be the way", then a bunch of influencers will retweet it as "anthropic kills writing tests", which some CEO's and team leads will parrot and get their team to absorb these practices even tho they are not the best practices. Some of these things are just reinventing the wheel, like this is a basic script called "run_simulator.sh" and a pre-commit test hook, not a skill and "commit gate" that needs to check session transcript: >To codify how to start the simulator or run your app, create a skill called verify. Put it in a .claude/skills/ folder in your repo, as close to the relevant code as you can, and add as many as you need. Claude knows to use the verify skill before committing changes. >We've also created a commit gate that can block a commit if the session did not contain /code-review or /verify. We've set it up with off, observe, and enforce modes. |
|
| ▲ | baxtr 3 hours ago | parent | prev | next [-] |
| Wait, are you saying a company that makes money from selling tokens generated a content piece recommending people solve their problems by using a lot of tokens? Scandalous. |
|
| ▲ | pluc 3 hours ago | parent | prev | next [-] |
| Here's how to properly use XYZ, yes it'll cost you more but we manufacture it so to us it's no issue. Do you not see the propaganda through the pseudo helpfulness? |
| |
| ▲ | hackyhacky 3 hours ago | parent | next [-] | | It's the same reason they give you a recipe for Chex Mix on the back of every box of Chex. | |
| ▲ | ben_w 2 hours ago | parent | prev [-] | | > Do you not see the propaganda through the pseudo helpfulness? I find it amusing that people phrase bog-standard "what can our product do" documents as "propaganda", even when it's a dogfooding exercise. Like, I guess? In the sense that all marketing can be seen as capitalist propaganda? | | |
| ▲ | pluc 2 hours ago | parent [-] | | Marketing is for products, propaganda is for agendas. | | |
|
|
|
| ▲ | simianwords 3 hours ago | parent | prev | next [-] |
| Its almost become a meme at this point: well actually you can achieve the same using X so why waste tokens?! The reason you should use Claude and not playwright always is because the the invariant you wanna capture is at the natural language level. Like imagine I wanted to verify that Claude Code works and doesn't break. I can describe the invariant I want in natural language. Like: make sure that subagents don't block the original thread. Doing this in playwright means that you have already bought into some previously existing architecture which means your tests are now correlated to your code. Having the invariants written in natural language allows you to decouple code vs the invariant. I expect a lot of greybeard conservative programmers to keep opposing using AI here until eventually, not using AI will be seen as more risky. |
| |
| ▲ | LelouBil 2 hours ago | parent | next [-] | | The better thing is do use both, use claude to convert the invariant to deterministic code, and review said code to be sure it matches your natural language idea of it. The issue with only having the natural language specification is that it is non-deterministic. | |
| ▲ | fingerlocks 2 hours ago | parent | prev [-] | | That’s a terrible example because you can verify that at runtime by attaching thread watchdog daemons. This is invariant across all languages natural or otherwise, invariant to the binary, invariant to tokens or context, invariant to network access, and you can set the assertion, making it invariant to the thing your trying to accomplish. I expect a lot of no-beard man-child programmer wannabes will brain rot early in their careers and find themselves wallowing in the hopeless despair of sitting in front of a computer arguing with a chatbot for decades on end, living a life trapped under the weight of endless mediocrity, unknowingly desperate for a modicum intellectual stimulation that you’ll never find because you can’t prompt for that, bro. |
|
|
| ▲ | SecretDreams 3 hours ago | parent | prev | next [-] |
| You don't get a nice juicy IPO valuation by trying to save money or be efficient with your token usage! |
|
| ▲ | catchnear4321 3 hours ago | parent | prev [-] |
| It behooves the token merchant to espouse the benefits of having more tokens. Not that it is an explicit, conscious direction, but it very much is one. You’re not wrong, in that free flowing tokens are enough to convince such a group it’s simply good advice. |
| |
| ▲ | adamddev1 3 hours ago | parent [-] | | The more you use AI to write your codebase, the more you need AI to manage your codebase. The more you use product X the more product X you need to buy. Nice to be selling product X. I'm not working for a manager, so I don't need to use it at all. | | |
| ▲ | catchnear4321 3 hours ago | parent [-] | | This feels a bit like the other extreme. It can meaningfully “type” faster than you. Take the handicap if you really wish to, I suppose. | | |
| ▲ | adamddev1 2 hours ago | parent [-] | | I do understand that it is extreme, but I think the principled stand makes my code and my skills better in the long run. I'm banking on building less, but building better. Otherwise where do you draw the line in losing quality and understanding? And for what I'm working on, typing is rarely the bottleneck anyways. I still use LLMs (cautiously) for searching and troubleshooting API things. | | |
| ▲ | RetroTechie 2 hours ago | parent | next [-] | | > I'm banking on building less, but building better Same here. But as opposed to the "use AI" vs. "don't use AI" crowds, I'd prefer a more pragmatic approach: Do whatever possible to maximize the quality of the software. And with quality I mean correctness, reliability. AI is just another tool there. Where it helps produce more correct, reliable code, consider using it. Where it doesn't, don't. Sadly it seems many companies have a different definition of "better" (featuritis, shortest time to market, developer productivity, etc). | |
| ▲ | catchnear4321 2 hours ago | parent | prev [-] | | Letting go of the typing isn’t a light decision. And it is harder to get good output when it’s also simply a lot of output. I’m banking on that being an evolving, but necessary skillset. For a while at least. And I’m still gaining experience in the process. Plenty of exposure, plenty of prior experience, it’s a weird mix of management-ish context handling and sme mentorship while also providing the research benefits you mentioned. |
|
|
|
|