Remix.run Logo
pixl97 4 hours ago

You still need to define assumptions somehow. What you want and what the model wants will not match up by default.

Madmallard 4 hours ago | parent [-]

In my experience telling it what I want is not a reliable process at all whatsoever if what I'm asking for is sufficiently complex, no matter what context I provide. So instead I break tasks down into very small parts, ask for solutions to those that I can reasonably quickly assess and then put them together myself. Asking it to do the architectural or deep algorithmic legwork IME wastes so much time and is often just wrong.

TeMPOraL 4 hours ago | parent | next [-]

There's a balance to be found here, that's unfortunately very hard to find at times.

In my experience, there are two classes of tasks: some are very "in-distribution", and for those LLMs can near-flawlessly perform the "architectural or deep algorithmic legwork", with maybe a single second round to fix the mistakes. For others, I have to break the tasks down myself, and often it's a "death through thousand papercuts", because the size of a task that I can quickly verify and the LLM will not screw up with > 50% probability is small enough that it's sometimes net negative time spent relative to doing it myself (and using LLMs only as glorified search engine and article summarizer).

I like to tell myself that I'm getting better at recognizing these two classes up front, but I'm still frequently surprised when "type 1" turns out to be "type 2".

But circling back to the main topic: with "type 2", agent instructions are paramount, if only to enforce the "small steps, pre-commit to scope and methodology, verification at the end, user doesn't even want to know about anything in between" rules, as agents naturally want to run ahead faster than I can keep up with.

verdverm 3 hours ago | parent | prev [-]

A good practice to use (ime) is having it do research for the larger task, propose alternatives, and write that in a file. You can then review and comment that up, go through another iteration.

Then when it comes to implementation time, things typically go much smoother for larger changesets. Be wary to not overplan, as we all know how often we realized we missed something once we get into the details. Here, I stop the session and go back to iterating on the design/plan doc. Not a step-by-step guide, if you don't instruct them to the difference, they will just pseudo-implement in the plan like they do in their thinking traces, need to be be explicit about the level of detail.