Remix.run Logo
▲ mzhaase 7 hours ago

Here is something I found this week: copies the entire db to memory, inserts a single row into in-memory db, deletes everything from real db, writes everything back. Sentence that caused this is "do XXX in one transaction".

All tests pass, everything seemed to work. Would not have noticed of I didn't read through the code. Amount of other stupid shit I found is crazy.

▲ricardobeat 7 hours ago | parent | next [-]

These anecdotes mean little without knowing context, model, harness, prompt. The technology has moved a lot in the past year, and does require some adjustment.

My very first experience with OpenCode led to a `rm -rf /` within minutes, which I caught by chance watching the output stream in. Yet 99% of my code is currently written through an agent.

▲xigoi 5 hours ago | parent [-]

Ah yes, the favorite response to any criticism of LLMs – you are using the wrong model.

▲XCSme 7 hours ago | parent | prev [-]

A performance benchmark would have easily caught this though, ask the LLM to create and run benchmarks and see how those change after each new feature/major change.

▲verdverm 6 hours ago | parent [-]

I have found test coverage reports to be really helpful as well.