Not my experience at all. I pointed frontier models at some PyTorch code, told it to make it faster, and it optimized it 10x fold.

You’re probably limiting the frontier models by being specific.

▲

vladms 5 hours ago | parent | next [-]

I am sure people could find code such that an LLM can optimize it 10x fold. I have seen many types of code in my life (well written, horribly written, fast, slow, etc.).

I do feel there is a limit to LLM-s today. I did not try, but I doubt it will work if I would tell it "make me a web browser that is bug free, perfectly secure, works as efficient as possible on my architecture and has best UX for me personally".

Knowing where that limit is, is as hard as it always was knowing how fast a team of engineers was going to do a project.

	▲	latentsea 4 hours ago \| parent \| next [-]
		> I do feel there is a limit to LLM-s today. I felt there were limits 4 years ago. I couldn't get even an 8k context window. It's starting to feel like the important gaps are the only ones left that need to be closed before there aren't any left. It also feels like next year they will start meaningfully closing.
	▲	throwuxiytayq 4 hours ago \| parent \| prev [-]
		> I did not try, but I doubt it will work if I would tell it "make me a web browser that is bug free, perfectly secure, works as efficient as possible on my architecture and has best UX for me personally". I do feel there is a limit to human-s today. I did not try, but I doubt it would work if I asked literally any programmer I know "make me a web browser that is bug free, perfectly secure, works as efficient as possible on my architecture and has best UX for me personally". Hell, I'm willing to bet they'd fail at this task even with an unconstrained snacks and kombucha budget. Humans have a long way to go. My job is safe.

▲

progval 5 hours ago | parent | prev | next [-]

Not every optimisation is the same. LLMs can optimize code that is behind known practices, but won't usually invent new data structures or algorithms to solve a unique problem.

▲

Zambyte 5 hours ago | parent [-]

While it's unlikely for LLMs to implement patterns not present in their training, current training techniques absolutely can produce practices and data structures and algorithms that humans have never done before. Modern LLMs are not just trained on text. They are trained on computer use, including - at a high level - solving programming problems with code. They are not just rewarded for producing the most likely next token. They are rewarded for producing tokens that solve problems. Patterns and techniques can emerge from this that are not present in any human behavior.

	▲	lifeisloving 4 hours ago \| parent [-]
		They are rewarded for code compiling, thats about it. Thats why the code is terrible. Ive yet to look at a project made by an llm whos user swore this "frontier model is the one", where the code wasnt utter garbage. The heavy llm users dont care as long as it compiles.

▲

lifeisloving 4 hours ago | parent | prev | next [-]

This doesnt even apply to his issue. What does PyTorch code have to do with complex concurrency or complex systems in general.

Theres lots of bad Pytorch code, just go ask george hotz. This isnt the magic you think it is. And nobody wants to hire the guy that just points llms at things and says make it faster. Things have value because talented humans make them. Its why a luxury coat is worth more than the linens that make it, or one from walmart made by a machine. This will 1000% apply to programmers. I think OP will be fine.

▲

mjr00 5 hours ago | parent | prev | next [-]

The fact that you're pointing it to PyTorch code in the first place is being specific.

Pointing a frontier model to a 10 or 100MLoC codebase and saying "make this code fast, make no mistakes" doesn't work. As an experiment I recently tried this with a relative small (500KLoC) codebase and it got stuck on believing that the primary cause of slowdown was the database not using a connection pool. (Which was completely irrelevant for this specific code.)

▲

xpct 3 hours ago | parent | prev | next [-]

With how you presented this, it makes it sound like it was either a toy codebase or something outside of your domain already.

▲

3 hours ago | parent | prev | next [-]

[deleted]

▲

bob1029 5 hours ago | parent | prev | next [-]

How large is your codebase?

▲

intended 5 hours ago | parent | prev [-]

The example provided covered both options.

In general the OP is right, the people who get the most value out of LLMs are veterans.