Remix.run Logo
simianwords 8 days ago

I think its a good idea but how do you not accidentally benchmark hack here?

GabrielBianconi 7 days ago | parent [-]

We set up dataset splits and the usual best practices. Of course, if you overdo things, you can still hack benchmarks; our goal isn't to publish SOTA numbers but rather to illustrate results from our methodology. We didn't even tune hyperparameters, we just used the default choices. Definitely a valid concern for teams chasing SOTA though.

Thanks!