"Baking in" a model into a chip is a bad idea because chips take 2 years to tape out and then you're stuck doing inference on llama 3 in 2026 when fable/sol are available. Every accelerator is a tradeoff between flexibility and performance and GPUs are already pareto-optimal

▲

guhcampos a minute ago | parent | next [-]

There are other options. I worked for a startup called NVXL and we were programming DNNs into FPGAs using OpenCL, on custom boards we built to plug into NVME. It worked great, but it wasn't fast enough at the time to compete with Nvidia, or even Intel AVX512. Ultimately the company failed, but maybe some hybrid like that could work for LLMs? I haven't been up to date on how DNNs and LLMs look like under the hood these days, but there's got to be someone doing something similar.

▲

twobitshifter 2 hours ago | parent | prev | next [-]

It depends when the good enough level hits. Pretty sure we are almost there for most common applications of AI.

▲

dgacmu an hour ago | parent | next [-]

That's only half the problem. OpenAI is contractually obligated, if you will, to believe that models will continue improving at an impressive rate for the foreseeable future (otherwise their valuation makes no sense).

If you believe that, then you should expect to get Sol-level performance out of a Luna-cost model within six months or a year. If you have a system with the weights baked in, that means you're going to end up serving that Sol-class model several times more expensively than it will take someone who comes along a few months later. (such as what recently happened with DeepSeek's update.)

And under that assumption of continuing advancement, baking things in doesn't make sense in general - it's a play you'd make if you think things are slowing down a lot. Which may be right but it's not OpenAI or anthropic's play.

	▲	adventured an hour ago \| parent [-]
		Assume a $800 billion valuation. $100 billion ad network. $30 billion op income. 26x price to op income ratio. It's right there for them to grab, or someone else to grab. Their valuation does make sense if you believe: 1) they can retain a massive user base and 2) a massive user base can be monetized. Future value is almost always pulled forward these days for high growth tech companies. An LLM the size of Google search in users is even more valuable than Google search. The ad market for LLMs will be even larger than search was (no matter what HN prefers). The monetization part is the easier part. Silicon Valley understands extraordinarily well how to build ad networks. If OpenAI maintain their gigantic user base, a $100 billion ad network is a given bolt-on. They'd have to screw that up in an epic way to not get there. Facebook - Insta - WhatsApp is an absolute dogshit tandem with a gigantic user base. $228 billion in ad sales and still expanding 10% per year. Google knows this is what's happening, that's why they don't care about chasing Anthropic very much. They're busy completely remaking how their core search business works.

▲

usef- an hour ago | parent | prev | next [-]

The whole point is that it's supposed to be more efficient. But models are also still getting absurdly more efficient every year. 18 months is a long time right now (and 18 is only time to tape out, not operational in data centers).

Even if they stopped get more efficient right now, you would also not be able to train them against new tools/harnesses or knowledge.

▲

adventured an hour ago | parent | prev [-]

The good enough level isn't ever arriving. We're in the first or second inning for LLMs. They will rapidly subdivide in complexity, they will not stagnate in the next decade.

Beyond the model, when would you freeze processor performance, such that it was good enough? Because that's exactly what freezing on Talaas is premised around.

The semiconductor technology will also continue to improve. You lose twice. Talaas is one of the dumbest ideas I've seen in semiconductors in decades.

▲

pantalaimon an hour ago | parent | prev [-]

Well we'll see those surplus chips being repurposed for toys then. Who wouldn't want a new Furby that can actually hold a conversation.

	▲	vunderba an hour ago \| parent [-]
		I've seen several attempts even on HN of the LLM meets Teddy Ruxpin (or more accurately AG Talking Bear) but most of them offloaded the AI to some off-site servers. I’d like to think that most parents would be weary of handing their children what basically amounts to a tape recorder that siphons all the data off to a large corporation. OTOH, a completely local one (LLM + VAD + Speech Rec) would be a fun little thing to build. https://en.wikipedia.org/wiki/AG_Bear