| ▲ | pupppet 8 hours ago |
| I don’t understand how there isn’t a website out there tracking this stuff already. |
|
| ▲ | copperx 5 hours ago | parent | next [-] |
| https://modelregression.com/ |
| |
| ▲ | hn8726 4 hours ago | parent [-] | | Haven't looked into how accurate the page is, but the list of regressions on the bottom looks terrifying, at first glance? | | |
| ▲ | copperx 4 hours ago | parent [-] | | Yes, it looks like regressions are frequent, but sometimes performance goes back to baseline quite fast. |
|
|
|
| ▲ | bpavuk 8 hours ago | parent | prev [-] |
| how the hell do we even track that? and before someone says... —"Benchmarks!" ...I'll tell that they can be gamed so easily, and they are on a consistent basis. |
| |
| ▲ | solarkraft 7 hours ago | parent [-] | | Sure they are, but do you think they are continuing training to improve a model after release without bumping the version number, presumably only to game the benchmarks? | | |
| ▲ | nananana9 an hour ago | parent [-] | | If you're willing to cheat, isn't it just a matter of grepping for the benchmark's question and pasting the solution in the chain of thought? |
|
|
