I don't really track issues, as I don't need to. Just a recent example "please extract the tabular data from this visual" and the model had incorrect aligned records in one column, so the IDs were off by 1 in the data.

I'm sure in 95% of cases it gets it right, but it didn't this time, and I'm not sure how to actually work around that fact.

Not an attack on your experience at all! I would would definitely counter that multimodal are still error prone and much better output is achieved using a tool like textract and then an LLM on the output data.