Remix.run Logo
mattmcal 3 hours ago

It's not just web chat, V4 Flash 7/31 suffers from a lot of pathological behavior in coding harnesses as well, e.g. infinite loops, hallucinations, premature termination, and invalid tool calls.

pimeys 3 hours ago | parent | next [-]

All of these flash models have this. You have to build your harness so that it deals with it. Infinite loops are solved by having an error message that says what to do differently on failure, invalid tool calls are solved by making the tool schema less strict and detect things in the runtime etc.

Hallucinations you can't fix. Gemini is a bit worse there than DeepSeek, but there's not much research on how to fix that. The only one is the CaMeL paper by Google, where you tag every prompt and result and then for every assistant response or tool call you first check where it got that data and error if you notice fabrication. This one is really annoying to implement.

With larger models the fabrication starts when the context grows or if you have too many tools, for flash models it's much earlier. We use the flash models for repetitive agentic tasks, where the prompt defines clearly what to do and how. The whole run is about 4-5 steps typically, and context size stays in the comfort zone.

CharlesW 3 hours ago | parent | prev | next [-]

FWIW, I haven’t experienced any of that using V4 Flash via DeepSeek in omp. What’s your coding harness and inference provider?

rpdillon an hour ago | parent [-]

Same. My side projects are coded almost exclusively with the Deepseek V4 Flash 07/31 in omp, and it recovers beautifully in every case. I'm using OpenCode Zen.

bendangelo 3 hours ago | parent | prev | next [-]

This never happens on the deepseek api. It’s always a different provider using lower quants.

K0IN 2 hours ago | parent | prev [-]

I have used the flash model for over 3b tokens and ofc. I saw some hallucinations and premature termination (I also get this on Astra - way more often than with deepseek v4 flash), but I never had a infinite loop (using the copilot as harness).