> that doesn't mean other kinds of AI are not AI

I kind of agree, though I find "AI" to be an almost useless term, in particular these days. I dislike the current state of discourse where AI is used to mean inhumanly large models, so I tend to use a specific term like LLMs instead of AI. But the point here wasn't whether this or that form of ML is in fact AI. The term "AI" was almost never used in mainstream media until a few years ago, when it suddenly it was all over the place https://trends.google.com/trends/explore?date=all&q=AI,Biden... and nearly without exception referring to very large generative text/image models. So when a news article in 2026 talks about AI without specifying that it's nothing to do with ChatGPT etc. (and in fact using a very different method requiring no large pre-trained model and no dependence on large companies), I find it highly misleading.

I'm not sure I agree with the extremely easy to verify part. How do you validate that your new test suite covers the exact same edge cases as your old?

But yes, LLMs are great at tests. I'm not doubting that an LLM migrated some legacy tests, or that it was a task taking a long time. I am doubting the way it was presented in the article, that this was taking a full team of engineers dedicated to nothing but this, 5 years to accomplish (and presumably already spent 4 million working on this, since the total estimate was 6 million, and they've been at it for 4 year).

I think some PR flack got ahold of the fact that an LLM wrapped up migrating some legacy tests that a team had been slowly chipping away at for 4 years, between their normal feature work, and were on track to finish in 5, then wrote it up like it was that teams entire focus instead of a piece of tech debt.

That makes far more sense to me than spending millions for a team of engineers dedicated to nothing but rewriting an existing test suite.

> Migrating a test suite is exactly the kind of work that LLM's excel at beause it's extremely easy to verify

I don't think that's true. All you see is all the test pass. You don't know that the tests still cover everything that they used to.

LLMs excel where there is an excellent test suite, and you ask them to modify the thing that the test suite tests, and forbid them from changing the tests.

How is it bad faith to question corporate marketing blog posts? That's not bad faith, that's table stakes for critical thinking.

We absolutely should be skeptical when an AI company makes big claims. The fact that the company the AI company is talking about also claims the same thing doesn't change that. OpenAI is getting awareness and marketing out of this, and I'm sure Asana is getting something out of it too.

And I'm not even saying OpenAI or Asana are necessarily lying. Asana might not find out for months or years that some tests had been rewritten poorly, and don't sufficiently test the thing they were supposed to test anymore. For example. If they truly had 5 years of work, then I find it hard to believe that in two weeks of the LLM churning, they had the time to review all the new tests. They spot-checked, at best.

Maybe everything is great. Maybe the LLM did a wonderful job, and this was awesome for Asana. But we have no idea, and we're unlikely to ever find out. Unless, of course, it's in Asana's interest from a marketing perspective to tell us.

(Not sure what the URLs you posted in your comment are supposed to prove. They're unrelated to the issue at hand.)

The comment I was replying to said

> But you _can_ verify if you had bothered

I think pointing out I can't is indeed fair.