| ▲ | johnnyApplePRNG 5 hours ago | |
I was wasting hours yesterday trying to get DeepSeek V4 Flash (with Qwen 3.8 27b as the vision agent, actually) to read sheet music to pass a Terminal Bench 3 benchmark and none of it was working... nothing... I changed models to gemma 31b, I tried OCR models... nothing could get it... And then I realized, wait a second... you're testing the harness not only against a difficult benchmarking problem, but it's one you're literally never going to use the coding harness for either, lol. I don't write programs that read or interact with sheet music and I never will. tl;dr Being frustrated that a "state of the art" vision model doesn't have perfect vision is a fools errand. It can read and extract information from screenshots and PDFs just fine (my setup). No need to worry about edge cases. | ||