Have you any example materials that show a harness that an agent can be pointed at an app with some sort of telemetry tool, the results it gained and the cost of doing so? Because my experience is the same as the parents - the LLM goes on massive tangents, and the more tangents it goes on the worse the results get.

▲

ndriscoll 5 hours ago | parent | next [-]

Not sure what materials your hoping for, but I've pointed codex at pprof application endpoints and just told it to identify hotspots and propose lowest hanging fruit/highest ROI items to fix, which I then approved it to open PRs for. Reduced application CPU usage by ~30% on loaded servers.

It's not magic. Fairly obvious oversights that I or someone else on the team could've found and fixed if we had looked, but it was never a business priority so we never did until I curiously spent 5 minutes asking it to do it for me.

▲

maccard 5 hours ago | parent [-]

Ideally a blog post on an open source project that shows harness, model, prompt and the before/after measurements plus the PR.

	▲	redsocksfan45 29 minutes ago \| parent [-]
		[dead]

▲

simonw 4 hours ago | parent | prev | next [-]

Tell the coding agent to install or build its own performance and profiling tools.

A few lines of Python with a time.time() call is enough to start iterating on performance improvements, and the really good models know how to use much more advanced existing tools.

▲

colechristensen 8 hours ago | parent | prev [-]

Ultimately the harness is me and the experience is like managing an unruly toddler. You have to pay attention to what it does and issue corrections. Skills and prompts and AGENTS.md do some, nested sets of agents do some, but over it all is me keeping track of what it's doing and needs to do.

You have to size the unit of work to its useful attention span, you have to have code architecture that is conducive to units of work, and you have to have tools like ticket managers, etc. that keep the big picture and smaller pictures in mind. Reverting from agile practices to more pre-planning whole project documentation and architecture decisions helps.

But in the end it's you. LLMs have their limits and need humans to direct them.

▲

maccard 7 hours ago | parent [-]

Gotcha, so no.

▲

bmurphy1976 3 hours ago | parent [-]

You don't need a "blog post." You need experience managing projects, tracking progress, coordinating conversations, mentoring less experienced engineers, etc. That experience comes with practice and time.

If you really need something to latch onto, there's plenty of educational material available about how to be an effective scrum master. Start there.

	▲	colechristensen 2 hours ago \| parent [-]
		Indeed. I'm saying the human inside the engineering loop is essential. If a person needs training, particularly if the set of high level advice I set out in a reply is rejected, a blog post isn't going to cut it anyway. Some AI skeptics paradoxically insist humans are irreplaceable and then get almost mad(?) when you discuss how humans are necessary in the loop. At least that's my interpretation of what is going on, it's hard to tell. When building with LLMs I'm mostly product manager and QA, the whole task is having the judgement to evaluate and correct course. You can share advice but it's not a copy-paste situation where one set of prompts/harnesses/whatever solve all your problems forever. They have to be adjusted for a particular situation.