I'm on Wayland and the cable is HDMI 2.1 ultra high speed, which means 8k. Xorg is already gone on Ubuntu 25.10.

The GPU and monitor combination has full 10-bit HDR in Windows. But in Linux it's stuck at 8bpp due to nVidia driver not having 10-bit RGB output.

I don’t think your problem is RGB instead of BGR. That’s just the compositor’s work area and your monitor never sees it (it includes an alpha channel). Have you tried KDE Plasma? It sounds like KWin uses 10-bit planes by default when available. Maybe Ubuntu’s compositor (Mutter?) doesn’t support 30 bit color or must be configured? Or maybe you need the nvidia driver >= 580.94.11 for VK_EXT_hdr_metadata (https://www.phoronix.com/news/NVIDIA-580.94.11-Linux-Driver)