> Tokens become cheaper than tool calls

The author observes that a call to GPT-5.6 Luna is only 4-5 orders of magnitude more expensive than grep, and then predicts that at current rates of progress, calling an LLM will soon be cheaper than a grep. I think this is a good time to invoke Stein's Law: "If something cannot go on forever, it will stop." These efficiency improvements won't continue forever. It's more likely that the per-call cost of high-quality, compiled software like grep will be a lower-bound that LLMs asymptotically approach, rather than a line that they blow past with perpetual exponential progress. (Barring a true breakthrough in something like quantum computing or room-temperature superconductors.)

Yeah I bumped on that too. If it's possible to make llms cheaper than current grep, then it is also almost certainly possible to make grep cheaper.

4-5 orders of magnitude is huge. Assuming an order of base 10, it's 10000x-100000x. So a call to grep may return in 1s on a typical PC. That means a GPT call takes equivalent energy of 10000-100000 PCs to do the same in 1s. That's a difference that can't be equalized with scaling. It would require a revolutionary breakthrough.

I also don't understand where the idea that frontier models are getting better efficiency comes from. The results are certainly improving, but that comes from feedback and multiplexing requests, which cost more.

LLM is spicy memoizing, so it can potentially be faster than a tool call. But people will spend a month tweaking and testing to ensure they have the level of determinism they need, which means it's more expensive, and that they should have used actual memoization in the first place.

A classic case of someone projecting out to infinity from just after the first bend of the S curve.

> The author observes that a call to GPT-5.6 Luna is only 4-5 orders of magnitude more expensive than grep, and then predicts that at current rates of progress, calling an LLM will soon be cheaper than a grep.

At some future point where LLM hardware is cheaper than simply running grep, then grep equivalent would benefit from those selfsame hardware improvements and be cheaper to run as well, probably still by the same ratio.

Well, think that statement through a bit:

Grep reads through the entire file looking for patterns.

An LLM scans its neural net (in ways that I don't understand) which is kinda-sorta like having a huge index.

You can improve over Grep if you have an index; and the LLM has an index.

Thus, it's plausible that an LLM can be more efficient at reading its neural net (IE, index) than Grep reading the whole file.

It might never beat out grep, but it could beat some more expensive to call tools, similar to how heuristics will often be faster than exact answers. Rust Analyzer can be slow at times, I could see an AI tool taking over a subset of its work.

If the providers can't find a way to nickel and dime us to death they will no longer provide the service

Right. And some hardware improvements will speed up both grep and Luna, which won't close the gap.

A few years is hardly forever and the state of the world here indicates a lot of low hanging fruit still exists.

An LLM can certainly be cheaper than grep, because it’s an approximation, while a grep is deterministic and must examine every byte in what can be a relatively complex state machine for a regex based grep. There are other scales to consider like the scale of your local hardware vs the highly multitenant and high end hardware of the hyper scale inference providers.

There are already high volume models for coding inference where the reasoning time is crazy low and cheap per token where it can build reasonably simple software so blindingly fast it isn’t implausible the bottleneck is the latency in tools and networks. I find them hard to use at times because I don’t have time to think through the next turn by the time it’s done.

Regardless I wouldn’t be surprised to see a world where tokens are so cheap it’s not worth metering them but charging licensing feels with meter tiers at the far horizons to prevent abuse, charge outliers. Subscription models already set this stage well.

The other side to consider is bountiful capacity will also drive tokens to near zero price. The data center build out is barely underway and as it materializes, as hardware efficiencies improve, as techniques and model science and technology improves, harnesses, methodologies , etc improve, the economics flip from load shedding to trying to keep the data centers utilized. The economics lead to the world where tokens are not a unit of measurement for cost for anyone other than the inference providers to manage their utilization.

There is no reason to assume that quantum computers would benefit LLMs in particular. Perhaps we could implement LLMs as analog circuits to save energy.

> calling an LLM will soon be cheaper than a grep

From a computational standpoint this is obviously nonsense, but from an attentional one I'm not so sure. It may already be more attentionally expensive to use grep in some cases, such the moment you need to remember a non standard arg. And if this applies for performing a simple http operations, then it certainly applies going up the complexity chain.

What if the tool is more advanced, like an optimizing compiler: `g++ -O3 -march=native -x c++ - <<EOF ... EOF`

If the compiler invocation is sufficiently slow, the llm could consider outputting a binary directly?

For all we know matrix multiplications are a faster way to generate optimized machine code than branchy sequential compiler code with tons of heuristics and passes.

> Barring a true breakthrough in something like quantum computing or room-temperature superconductors

Won't that also help grep and then move the asymptote down more?

room-temperature superconductors, sure, but I fail to see how quantum computing will disrupt – in the medium term (25 years or so) – classical computing in any meaningful way

Is running LLMs (or some other ML workload) on/with quantum computers expected to bring efficiency gains?

The cost for my employer to employ me is only 4-5 orders of magnitude over that of the snack in the vending machine…

But grep is just a tool in a pipeline between a question in your brain and an answer you are searching for.

what if an LLM finds the answer early?

grep might continue to read everything, doing the wrong thing correctly.

"Did you know that disco record sales were up 400% for the year ending 1976, if these trends continue...AY!"

I think it's an interesting thought experiment: could an LLM call be a "cheaper" grep?

Especially for deterministic activities it just feels impossible to imagine general LLM tech handling the problem better, despite everything being said.

But hey, tech is filled with "smashing the generalist hammer works better than the specialized tooling". Would be odd though!