Remix.run Logo
meowface an hour ago

At this point I barely put any value in any of the benchmarks. I just use the models for coding (and related things like software product design/planning/ideation/etc.) tasks and judge them subjectively, and also see how others judge them subjectively on HN and Twitter.

girvo an hour ago | parent [-]

I use benchmarks…

…that are my own private internal suite on my own code bases where I can judge the output properly

I also measure wall clock time to completion which has been a surprising separator in practice.