As people have mentioned, it isn't that surprising that a specially trained model beats a general purpose model.

However, I think it kinda misses the potential of llms for diagnosis. What happens if we also include journal notes? What if we let the model generate questions to ask the patient and include those? What if the images aren't perfect examples and happen to contain out of distribution content like a tattoo?

In other words… more dimensions for a bigger set of vectors?