> But I also think the demand for "fast/cheap/good-enough" models is just about to take off.

There's a sort of "revelation" I had in ~early '24 when I used a 7B local model with a library called Guidance (initially out of MS, then the team moved) to create a flow where the model would receive pseudocode for tests, first write the tests, and once I approved then started writing code until the tests passed. This was before "thinking" models, and yet using that library I was able to "guide" the model in the required "prompt / instruct" context such that it was working towards completion, and I saw the first things like we see now in the thinking traces "oh, test x doesn't pass because blah, I need to..." and so on.

Anyway, the revelation was "even if the models never improve, I'll have years of fun finding out all the ways I can use these things". And, obviously, the models improved a lot since then. But I think that revelation can still be applied, as a sort of "truism". We have, right now, access to things that 10-20 years ago would be considered magic. We are still finding ways of cobbling together systems with glue, duct tape and prayers and find new things they can do.

I think the "good-enough" stage has come not just for API models (cheap, fast, etc) but for local as well. Even if slower, even if clunkier, but they are good enough for a set of ever increasing tasks, and what's more it's incredibly fun to work with them.

Yes.

The infancy phase of this technology is represented by the pursuit of making wildly grand, wildly expensive, all-purpose models that somehow discern a user's full accurate intent from a lazy, underdeveloped, vague idea that they ambiguously and poorly express in a couple dozen words.

The adolescence will arrive as those outsized and ill-considered ambitions collapse and we instead see a cambrian explosion of restrained but efficient model+harness-tuples that have been distilled, finetuned, and rigged to deliver on narrowly scoped but idiosyncratically-shaped tasks with incredible efficiency and erogonomics.

To me, most local models work just fine for anything you can be patient for. If I want something quicker, I will go to a SOTA model via API, but with multiple 3090s, I have never really needed a hosted model for a lot of my experiments.

For code, they are great, but for creativity for NPC controllers, they leave something to be desired, but work well enough for testing, so I don't burn tokens until I'm actually playing my games.

But nothing one-shots a prototype better than Fable 5. I can have a prototype built in 30 minutes, hooked up to my local LLMs and Claude Code is very good at testing the interactions and even tuning the prompts of the NPCs for better experiences.

Yep, I've been having excellent experiences with the models even from the 2023 era. They required a lot of "holding it right" (mostly: being very precise in what went into the context) but their raw coding capabilities were astonishingly good even then.

However, back then I was getting the AI to write individual functions or classes or a test suite. I was decomposing the larger task into smaller tasks, delegating some of them to the AI, reviewing the results and composing the codebase from those. I was also essentially the harness.

Today the models can write and test and deploy an entire project. In terms of the code quality, I actually don't think today's frontier models would have written it much better than the 2023 models did. So in terms of raw coding capabilities i.e. converting a high-level specification into working code, I think we hit the peak way back in 2024 itself.

What has changed is the AI has learned how to do the task I was doing (besides being the "harness"!), which was the mid-to-higher level "engineering" aspects like decomposing a task, specifying it to a reasonable level, reviewing the outputs, and course correcting as needed.

I'm not sure if that is something the AI labs explicitly focused on during training (which may be why Meta is having its highly paid engineers do annotation work), or an emergent property of "better reasoning" (which I believe Dario implied in a podcast), or some mix of both.

But the fact remains that even the weaker models are more capable than we realize, and many being open weights, are here to stay.

While they are improving rapidly, or as you say even if they don't. The next stage is for hardware companies ( cough Apple cough ) to ship these Local Model ready hardware in their products.

It will be interesting to track the improvements of these 7B model over time.

There will be a turning point in the next few years where it attract enough consumer attention to create yet another Smartphone and PC super cycle.

> We have, right now, access to things that 10-20 years ago would be considered magic

These things would be considered magic even 4 years ago!

There's A LOT low hanging fruit still out there for sure. And with antigenic systems being able to do the boring repetitive work of looking for that low hanging fruit I think we will see interesting things indeed. Also I think heuristics is where its at for such things. Once you describe some good heutistical structures for the research models to always follow related to "creativity" and such things, thats where we will see biggest difference. The agentic systems know the scientific method well and can follow it they just need the ability to be "creative" so their sampling becomes less rigid.

I see it in a slightly opposite way: even the good models are relatively cheap, and so I worry what we might miss by spending too much time playing with the Sonnets of the world when the Opuses are still objectively a bargain for the power they bring.

I think there's something subtle about language and ambiguity that means they aren't designed to become superintelligent autonomous machines. They're value is as information repositories that actual intelligent autonomous machines (us) mine and string together.

Just being able to instantly generate a complicated query expression to pull specific bits out of a JSON blob sold me. It's awesome that I can ask Claude to build a whole feature and it will often one-shot it for me, but generating utility bash / python scripts or little throwaway utility webapps is what really excites me.

Same. Mistral 7b has been more than I ever needed for text for years now.

Unless you must 1-shot with no harness it’s the same amount of power, maybe more because the big “good” models make too many assumptions and tend to become rigid.

Mistral 7b can do anything, and it’s basically instant even on an M3

If someone has an old GPU laying around, say a GTX 1080 with 8 GB of memory, would that be enough to get a (small?) local model running?

BTW structured/constrained generation has so many places to trivially enable jailbreaking/alignment/safety problems that closed source models heavily limit the full expresivity of grammars and capabilities, particular of on-the-fly dynamic grammar construction/reconstruction.

ppl keep talking about the supposed unexplored and untapped "model overhang" but very few things in the world are where you can write elaborate test criteria to before using ai.

A sales person sending a prospect email doesnt have a way to write a test harness for it. Yet these tasks dominate what humans do compared to writing a crud app . otherwise anthropic wouldnt have trillions dollar valuation

I've amassed access to 4 different GPU rigs with 128GB to 72GB; I didn't this before I event touched an agentic engineering harness. It was sometime in February/March when I set them to first tackle small problems, and now with deer-flow, they're scaffolding full project/scope implementation and I'm finishing off the fine details around the problematic edges.