Is the load balancing of the relays out of scope? It doesn't seem to be addressed in the write up unless I missed it.

kixelated 4 days ago | parent | next [-]

EDIT: Sorry I just noticed this was directed to Cloudflare. They're using the same architecture as Cloudflare Realtime, their WebRTC offering.

`relay.moq.dev` currently uses GeoDNS to route to the closest edge. I'd like to use anycast like Cloudflare (and QUIC's preferred_address), but cloud offerings for anycast + UDP are limited.

The relays nodes currently form a mesh network and gossip origins between themselves. I used to work at Twitch on the CDN team so I'd like to eventually add tiers, but it's overkill with near zero users.

The moq-relay and terraform code is all open source if you're super curious.

▲

nonane 4 days ago | parent | next [-]

Home much success have you have with GeoDNS? We've seen it fail when users are using privacy respecting resolvers like 1.1.1.1. It gets the continent right but fails on city/state level.

	▲	kixelated 4 days ago \| parent [-]
		It works well right now because there's only one edge per continent. But if I had traffic, anycast is definitely the way to go.

▲

bushbaba 4 days ago | parent | prev [-]

Anycast can have serious reliability challenges. It was common at GCP for a small QPS user of anycast to have their Load Balancers nuked in a given pop as it was backed by a single machine. But BGP showed it as still the best route. The major DNS based offerings don't have such issues.

	▲	kixelated 4 days ago \| parent \| next [-]
		QUIC has support for preferred address, where anycast is used for the QUIC handshake then the connection migrates to a unicast address. It still has issues but it's nice to have sticky established connections and avoid flapping mid connection.
	▲	wbl 4 days ago \| parent \| prev [-]
		I work for a CDN that does DNS steering. DNS record lifetimes are nonzero and can be surprisingly long. But you do get some very fine control over where data goes if resolvers cooperate.

▲

englishm 4 days ago | parent | prev [-]

I plan to cover more of the internal implementation details at a future date, possibly at a conference this fall..

But I can at least say that we use anycast to route to a network-proximal colo.