It bothers me that you have to explicitly state this to the agent. Makes me think what else is missing from that file which also needs to be explicitly stated, but I don't know what don't know. "Do a good job"?

This is where the harness shines. It's why products like Cursor have been growing like crazy. If you drop an extremely intelligent person into a problem they've never seen before, the initial context and knowledge-sharing just to get them up to speed is immense. They'll make a lot of mistakes until they're ready to fix your problem. The harness can leapfrog most of that. There is a booming industry in harnesses with no saturation in sight. We're not even close to maximising the utility of current models, let alone the amazing improvements which are happening week to week.

An agent doesn’t know what “job” it’s being asked to do until you tell it.

Imagine you’re blind and deaf and have temporary retrograde amnesia. You “wake up” one moment with a memory of some words in your head like “what is the bug?” …but you don’t recall the context of that question, and nor can you look/listen around to observe the context.

So you don’t know whether you’re e.g. at the office, in front of your computer, in the middle of doing some pair-programming (where, yes, you’d in investigate the bug thoroughly with tools), vs. having a conversation with a colleague over lunch (where the expectation is for you to tap into your knowledge + intuitions to either guess or say you don’t know — not to pull out your laptop.

That’s what it’s like to be one of these LLMs being prompted by some agent harness. Unless the harness injects the proper context into its “recent memory”, it just doesn’t know.

If you ask Fable or 5.6 Sol to improve performance, it will generally know to build a benchmark and create a test corpus. I'm not sure where the contrary suggestion is coming from.

Additionally, if you are asking agents to write test, be very careful. They will write tautology tests.

They will mock things to no end. They will flat out REMOVE assertions (saying it's not needed). They can also write test to assert the wrong result.

You have to always review it, it's exhausting.

Will, it depends on the AI. Anthropic used to have a lecture-length system prompt for their models to explain this stuff—part of the secret sauce for Claude Code—and famously found that the 5 series models no longer need it.

As usual, if you use anything but the best model available I’m going to state that the better ones do better. If you do use the best model available, then I’ll just mention that Fable still has limits and still needs some guidance.

One thing it does not do is deliberately build tests which test nothing at all, or which restate the code under test. I mention this because certain other models absolutely would.