I’m so confused. What is the difference between a peer relay and a DERP server that is self hosted?

The issue I have is I’m trying to connect two devices where one is behind a CGNAT that always causes the connection to be relayed even though the other one is not behind a cgnat with proper port forwarding. Would a peer relay solve this but is it like a DERp where I have to host it on a VPS separate from my existing two networks or is this something different where I can host the peer relay on the network not behind a CGNAT and somehow it will link the two networks through it?

You should be able to stand up a peer relay on an existing tailscale device - so your proposal is correct! Try setting one of the devices up as a peer relay per the docs here: https://tailscale.com/docs/features/peer-relay