> I used Inception v3 pretrained on everyday photographs and then fine-tuned all layers on 68,011 dermoscopic and clinical images from the ISIC Archive on a single H100 GPU.

>These 200 test lesions, and every other photograph of the same lesion or patient, were removed from its data (including the validation set) before training, so it had never seen them, and they were not used to choose the model or set its threshold.

This is just benchmaxxing. A benchmaxxed model will always beat an unbenchmaxxed model, but it will be brittle.

You can always beat a specific dataset with a small model by finetuning on a train/test split. But it doesn't generalize well to 'wild' images, because they are out-of-domain.

Training a task-specific small model is just good developer practice.

That's just good developer practice, not "benchmaxxing".

Benchmaxxing is when someone presents a model as being generally smart, but actually quietly overtrained it on some benchmarks that were supposed to be proxies for general information.

If you train a small model on a specific task that's not "benchmaxxing" because there's no deception involved. It's just training a model for a task.

> This is just benchmaxxing.

I believe the normal term is "gaming a benchmark test".

Nothing in what you quoted is related to that. It's a normal way to train and test a classifier.

The only way to determine if it's brittle is to test on a different dataset and look at the sensitivity and specificity.

Even so, there is nothing wrong with a model that is trained to a narrow domain when it doesn't need to deal with "wild" images.

What would be an unbenchmaxxed model here? A general image classifier like Inception v3, without the fine-tuning?

The 200 test images here are 'wild'. Fine-tuning in-domain is the whole point. This is what model training is. A skin cancer diagnosis tool doesn't also need to generalise to dog breeds to be valid, and will almost always perform worse if it does.

This is standard practice.

I developed many models from scratch deployed in various fields.

When you want to automatically detect a pathology from a give scan, you want it to be model that you own, train, use, and update, and not a generic vision capable LLM.

This is NOT what benchmaxxing is. You want the model to be accurate in terms of your cared metric- depending on the problem.

This is a good point and was an issue for skin cancer AI in the early days but has largely been solved now - newer models use ensembles of vision transformers and much greater number of training images and have been proven accurate in clinical trials. See for example https://skin-analytics.com/ (I am not linked in any way)

Could you explain why this is benchmaxxing?

Being good at the intended application is not benchmaxxing.