Training a task-specific small model is just good developer practice.

That's just good developer practice, not "benchmaxxing".

Benchmaxxing is when someone presents a model as being generally smart, but actually quietly overtrained it on some benchmarks that were supposed to be proxies for general information.

If you train a small model on a specific task that's not "benchmaxxing" because there's no deception involved. It's just training a model for a task.

TIL: benchmaxxing, aka overfitting?

It is benchmaxxing because the test distribution in these kind of image dataset generally matches the training much more than real world image. It used to be the biggest issue in real AI in most companies that the test data and real data is either different or risk getting drifted apart.

So it is not solving a task, it is solving the same test set. You can easily see these in simple OCR datasets like mnist. Draw a number yourself. It would be much less accurate than the 99% that good architecture claims.