I don't know why I should have a preference when it comes to the memory layout of an image. If metadata tells be to orient it this way, I'll orient it this way, if that way, then that way. It seems weird and wasteful to me (and potentially prone to error) to want to reorder the data in order to change the display orientation.

I can only imagine caring when it comes to streaming a very large image serially, and then I still have no idea what preference I would have (other than that I get the metadata first.)

As an analogy, if I have some irreplaceable video that has a bad aspect ratio, I fix it in the metadata, I don't transcode it. The data is a blob, the metadata is what tells me how to display it (even if it's just the mimetype.)