Then the question becomes: to what extent do you trust Xen and Qubes RPC?

I do have to somewhat trust Xen, but Qubes' isolation relies on hardware virtualization (VT-d), which statistically has much less security issues than Xen itself. Most Xen advisories do not affect Qubes: https://www.qubes-os.org/security/xsa/