i'm not sure why people expect agents to one shot everything to perfection with just a prompt.

There's a reason why we talk about software development lifecycle, design, architecture, testing ... It's because it's been the most reliable way to build and ship software. We shouldn't expect discard this and expect agents to perform well outside of this.

I'm treating LLM agents as junior devs who happen to have vast knowledge of software engineering. As their team leader i make them go through planning, implementation, bug sweeping cycles using strict workflows. And it works quite well, i've been working on several large projects (1M+ LOC java,typescript,c/c++) and by any measure the projects are healthy. Sure the code isn't that beautiful, sure i'd have written things differently but it's pretty good nonetheless.

Shameless plug here: i've been also working on https://kodfactory.com, the code factory i've built to work on these large projects with workflows, reviews, etc ... I'm cleaning things up to open source it later.

▲

0x457 an hour ago | parent | next [-]

> i'm not sure why people expect agents to one shot everything to perfection with just a prompt.

because that's how agents are marketed.

▲

44 minutes ago | parent | next [-]

[deleted]

▲

vanuatu 39 minutes ago | parent | prev | next [-]

we should exercise critical thinking then

heaps of people on this site expect them to be omnipotent then claim it’s fake when it doesn’t read minds

▲

uproarchat an hour ago | parent | prev [-]

I've never seen model providers marketing like that. What examples have you seen?

▲

0x457 an hour ago | parent | next [-]

Literally any coding agent marking material:

- https://cognition.com/

- https://openai.com/index/introducing-the-codex-app/

- https://www.anthropic.com/news/claude-3-7-sonnet

anthropic specifically brags about how good claude code is every annoucement of a new model. I will surrender that none of them claim its "to perfection", but IMO its implied because no one would claim that their model one-shots any issue to dog shit quality.

▲

infinite_spin 41 minutes ago | parent | next [-]

- https://openai.com/index/introducing-the-codex-app/

no where does this document suggest that codex can "one shot everything to perfection with just a prompt". It describes using a prompt plus agent skills (which are essentially many other prompts) to develop a playable game.. nothing about it being perfect or anything more than being in a playable state.

▲

hombre_fatal an hour ago | parent | prev [-]

Seems like motte and bailey fallacy. They say their models are good (the motte), therefore their models must one-shot everything to perfection (the bailey).

Besides, other people's claims about something doesn't give you license to abandon all critical thinking. Though it's evident they don't claim what you say they are.

▲

0x457 43 minutes ago | parent [-]

That's irrelevant. Questions was why people assume somthing, and answer is because that's how it advertised.

To be clear that's not what I'm thinking, even Fable 5 produces some hilariously bad results under some conditions and sonnet 5 produced great results under others.

	▲	hombre_fatal 40 minutes ago \| parent [-]
		> because that's how it advertised. But you didn't provide the evidence for that. You shared some links and then admitted they didn't claim it. It kinda seems like "because I think they're a little too positive about their product, I can set my expectations to anything I want and la-la-la it's their fault." And I don't see the problem with agents building test scaffolding as they go. It might be too defensive at times, like testing a shell script you don't run often, but big deal. It's kinda cool imo, and it's trivial to make it stop.

▲

Lalabadie an hour ago | parent | prev [-]

I don't really think they advertise "Create your app idea in one weekend night" and assume the general public will mentally add "... but hire an experienced developer to supervise the process".

▲

JoshTriplett 10 minutes ago | parent | prev | next [-]

> i'm not sure why people expect agents to one shot everything to perfection with just a prompt.

Every time you see a benchmark for "how long the agent can go without asking for human intervention", that's encouraging vibe coding.

▲

redox99 34 minutes ago | parent | prev | next [-]

> i'm not sure why people expect agents to one shot everything to perfection with just a prompt.

because that's the end goal? and for simple small stuff they're already there?

▲

perarneng 29 minutes ago | parent | prev | next [-]

If you explicitly ask the agent to make the perfect architecture for the problem and write it down in to a spec and have the developer agents follow it they will. Its just that coding agents have a hard time coding at think about architecture at the same time.

▲

jimmaswell 37 minutes ago | parent | prev [-]

> i'm not sure why people expect agents to one shot everything to perfection with just a prompt.

They do often enough that it's not a surprising event, depending on prompt quality, context available, ability for the result to be objectively judged and iterate on by the agent, etc. For frontiers on very high settings at least.