At some point I imagine they will be writing their own software for the entire training and inference stack. Kernels, libraries, why not? With compute so constrained it must be beneficial to optimize everything as much as possible

▲

stingraycharles 4 hours ago | parent | next [-]

I’m not buying this argument, the apps of the LLM providers are some of the most wasteful apps that exist. Have you ever looked at how much memory / CPU Claude and Codex use? They ship with entire virtual machine hypervisors for sandboxing.

I think it’s simply a matter of dogfooding and new feature development, they do have much higher velocity than companies of equivalent size and an unlimited token budget helps as well.

▲

brrwind 3 hours ago | parent [-]

This is completely orthogonal to the inference and training stack that the parent comment mentions

▲

foltik 3 hours ago | parent [-]

If it’s written by AI it’s gonna be the same overcomplicated overfit slop in one way or another. Performance pressure doesn’t fix that.

	▲	aurareturn 2 hours ago \| parent \| next [-]
		Depends on what the goal is and the skill of the engineer right? Give a verifiable benchmark for the AI agent to hit such as reduce token processing time by 5% without affecting output.
	▲	stingraycharles 2 hours ago \| parent \| prev [-]
		Assuming it’s entirely unsupervised, yes, which I can only assume to be the case at these volumes (hence the dogfooding). I do believe this will force the AI labs to eventually solve the bloat problem with AI, but we’re definitely not there yet.

▲

steve1977 4 hours ago | parent | prev | next [-]

Looking at their desktop apps, I don't think they're into optimizing.

▲

IanCal 7 minutes ago | parent | next [-]

The cost of running the apps to you is much less of a concern compared the cost for them running their own training.

	▲	steve1977 2 minutes ago \| parent [-]
		But shouldn't it be easy for their great models to fix it?

▲

trollbridge 3 hours ago | parent | prev [-]

They certainly are spewing forth large amounts of code, though.

▲

noosphr 3 hours ago | parent | prev [-]

I'm doing that now with gemini argon.

It's better than the ductape and glue pipelines that we had previous to this.

The software is the definition of plumbing no one cares about and testing it is trivial.