LLMs are like lossy compression of ~all of written text ever produced, with useful recall. To the extent that the corpus contains labelled examples of the given classification task, it's not unreasonable to think that we'll be able to build a decoder for that, just like we already have a useful decoder for next-token prediction. Extend to image classification the same way we already have multimodal LLMs.