You know how everyone thinks agents are bad at the thing they're good at and good at the thing they're bad at? It turns out I must be bad at testing, because I thought they do a reasonable job.

Most developers are not good at testing. That's why the role of test engineer exists.

I've seen in multiple projects things like assertTrue(true).

I'm sure the agent is better in testing than average enterprise developer.

I don't think these results show that.

I think they show that additional prompting for testing approaches have wide differences in error rate (worst has twice the error rate of the best) but actually no extra prompting is pretty fine and most custom prompts are worse than no prompt.

I thought so too but it's probably because I never review the tests. I made an exception recently and found tons of

    assert(CONSTANT_CONFIG == valueOfConfig)

    assert(prompt.includes("repo url"))