| ▲ | samuria 3 hours ago |
| It’s frustrating to create tests with models, not sure if it’s everyone’s experience, but every time I asked it to create tests, it makes the tests conform to the sometimes broken code. Rather than calling out what the logic/code is wrong, it will make the tests conform to whatever is already broken. |
|
| ▲ | forgotmypw17 3 hours ago | parent | next [-] |
| I recommend iterating on your prompts. Prod your agent first until it can explain back to you what's wrong with the tests. Then ask your agent to write a better prompt for writing tests (and save it to an .md file) Then reference that .md file and iterate on the tests again. It may take a couple of tries, but I've had good results with this strategy. Ended up with a whole prompting framework. |
| |
| ▲ | doginasuit an hour ago | parent | next [-] | | The problem I've had is even when it can explain back the issue, it still writes for itself awful instructions with extra fluff, strange points of focus, and borderline misinterpretations of what I said. Maybe that's not a big deal, they can still be generally useful, but they seem important to commit with the project if they will be used going forward. I hate the idea of a massive amount of slop riding along with the code, and slop begets slop. The idea I'm chasing now is writing the documents myself with the hope that enough examples and a document about writing documents will be enough to allow it to write clearly. | |
| ▲ | skydhash 2 hours ago | parent | prev [-] | | Prodding an agent vs doing interesting things with my time? What should I choose? To be fair, some people like to prod. But, for me, I'd rather take hours reading manuals than do trial and error prodding for 30 minutes (even if I knew it would be that short). Yes it's not optimal in terms of time, but it's better for my soul and well being. | | |
| ▲ | HiPhish an hour ago | parent | next [-] | | > Yes it's not optimal in terms of time, but it's better for my soul and well being. It is also optimal in terms of time in the long run. The prodder will have to prod over and over again because he has learned nothing. You on the other hand will actually understand what is going on and will be able to either do it yourself or give precise instructions to the machine next time. Don't let yourself become a meat proxy. | |
| ▲ | 0xcoffee 2 hours ago | parent | prev [-] | | It's a good use of your time to ask the agent to reflect and update it's skills if you intervene to manually steer it. Why wouldn't you want it to learn from your guidance? |
|
|
|
| ▲ | Waterluvian 3 hours ago | parent | prev | next [-] |
| It can’t magically know expectations. If the code is well documented or there’s decent associated documentation, I’ve found it has no issue writing tests. |
|
| ▲ | hulitu 2 hours ago | parent | prev [-] |
| > Rather than calling out what the logic/code is wrong, it will make the tests conform to whatever is already broken. It is the same like humans: if they say your code is wrong, you will hire someone else who doesn't find so many bugs. It doesn't look good. |