I've had lots of success with generating coordinates and answering questions using the UI-TARS model https://github.com/bytedance/UI-TARS.

I’d also checkout midscene, you can set the model and UI-TARS works but you can also use qwen vision models and it works.