If you're limited to open source models, that's very true. But for larger models and depending on your document needs, we're definitely seeing very high accuracy (95%-99%) for direct to json extraction (no markdown in between step) with our solution at https://doctly.ai.

In addition, gemini Pro 2.5 does really well with bounding boxes, but yeah not open source :(