The existing WebTransport API implemented in all browsers actually supports you providing the fingerprint of a certificate that can be self-signed.