> By that metric, everything is gameable

Usually in cases like this you would use a testing set created after the model was trained.