I've tried nanonets but it seems very sensitive to the prompt, changing it slightly turned the output to rubbish. When it worked it was pretty good.

This is true. It’s not meant to be run with any prompt but the one they trained with. I found that out as well. It’s only meant for ocr. Qwen 2.5vl is better if you need that option.