But other than it surviving, there's no way to define fitness.

It's more reproducing than surviving. If the population of some species increases and the number of copies of some allele remains constant we could consider than gene less fit than the other alleles, in the population genetics sense. So it's frequency rather than survival that geneticists look at. But that proves that there are indeed other ways that they could have defined fitness if they wanted to.

That is correct. Biology uses the term fit slightly different than the general public