> There's so much work in delivering products that will carry your brand, and then must be supported.

People think otherwise with AI partly because Anthropic kept telling us that they didn't have to write code or review code any more for most of their work. Their agent swarms just comb through their github, slack and wikis to figure out what to do next, and another swarm of agents just review, test, merge, deploy, A/B test, and revert the code. Boris alone merged nearly 300 PRs in the past week (or two?). So the top research labs seem have broken the productivity seal.

And then they talk about this recursively self-improving AI that is so powerful, so autonomous that they advocate that every company should be prepared to "pause" the effort. And their Fable/Mythos has this specific restriction as mentioned in their model card[1] that they are going to reject requests to tune and train models because, well you guess it, the models are too powerful to be used by mere mortals.

[1] We’ve implemented new interventions that limit Claude’s effectiveness for requests targeting frontier LLM development (for example, on building pretraining pipelines, distributed training infrastructure, or ML accelerator design). Using Claude to develop competing models already violates our Terms of Service, but enforcing this restriction through our safeguards avoids accelerating the actors most willing to violate these terms. Unlike our interventions for cybersecurity, biology and chemistry, and distillation attempts, these safeguards will not be visible to the user. Fable 5 will not fall back to a different model. Instead, the safeguards will limit effectiveness through methods such as prompt modification, steering vectors, or parameter-efficient fine-tuning (PEFT).

I think taking Anthropic or any company in this space at face value is naive at best though. AGI has been 6 months away for years now. Surely anyone can think this through: Anthropic knows what theyre doing with their public facing repositories, they know to make things enabled by their tech seem impressive. I would consider Bun etc. examples of this.

Realistically, nobody intellectually honest really knows.

People think otherwise with AI partly because Anthropic kept telling us that they didn't have to write code or review code any more for most of their work.

Even if that were 100% true, it only collapses the coding effort to near zero. Anyone who's built and shipped a real product should know that coding is maybe 50% of the work, and on a mature product it can be much less.

> People think otherwise with AI partly because Anthropic kept telling us that they didn't have to write code or review code any more for most of their work. Their agent swarms just comb through their github, slack and wikis to figure out what to do next, and another swarm of agents just review, test, merge, deploy, A/B test, and revert the code. Boris alone merged nearly 300 PRs in the past week (or two?).

Apart from many other issues with this, heavily subsidized subscription plans won't last forever, and if you start burning your own money on tokens in this way, you'll soon realize it's terribly inefficient.

I’ve been wondering if “you’re not google” when learning about googles software dev process applies to Anthropic. Anthropic is a company that A. Has cheap unlimited access to its models and B. Is probably largely insulated from the types of tradeoffs that the rest of industry has had to observe in the post-ZIRP era.

Like did they break through the productivity seal? Or are they willing to spend that much more on it since they see their failure as a like existential threat to humanity. I doubt it our boss sees your software the same way.

> the safeguards will limit effectiveness through methods such as prompt modification, steering vectors, or parameter-efficient fine-tuning (PEFT)

Holy crap that is dark. I like learning about ML for fun, and now I have to assume that their model is intentionally misinforming me to sabotage my learning? It is absolutely bananas that somebody decided that was ok behavior.

Anthropic is full of shit.

[dead]