> Path MTU discovery has not been enthusiastically embraced

Ugh. I don't understand this. Especially passive PMTUD should just be rolled out everywhere. On Linux it still defaults to disabled! https://sourcegraph.com/search?q=context%3Aglobal+repo%3A%5E...

▲

whiatp 8 months ago | parent | next [-]

PMTU just doesn't feel reliable to me because of poorly behaved boxes in the middle. The worst offender I've had to deal with was AWS Transit Gateway, which just doesn't bother sending ICMP too big messages. The second worst offender is, IMO (data center and ISP) routers that generate ICMP replies in their CPU, meaning large packets hit a rate limited exception punt path out of the switch ASIC over to the cheapest CPU they could find to put in the box. If too many people are hitting that path at the same time, (maybe) no reply for you.

More rare cases, but really frustrating to debug was when we had an L2 switch in the path with lower MTU than the routers it was joining together. Without an IP level stack, there is no generation of ICMP messages and that thing just ate larger packets. The even stranger case was when there was a Linux box doing forwarding that had segment offload left on. It was taking in several 1500 byte TCP packets from one side, smashing them into ~9000 byte monsters, and then tried to send those over a VPNish network interface that absolutely couldn't handle that. Even if the network in the middle bothered to generate the ICMP too big message, the source would have been thoroughly confused because it never sent anything over 1500.

	▲	toast0 8 months ago \| parent \| next [-]
		> The even stranger case was when there was a Linux box doing forwarding that had segment offload left on. It was taking in several 1500 byte TCP packets from one side, smashing them into ~9000 byte monsters, and then tried to send those over a VPNish network interface that absolutely couldn't handle that. Even if the network in the middle bothered to generate the ICMP too big message, the source would have been thoroughly confused because it never sent anything over 1500. This is an old Linux tcp offloading bug; large receive offload smooshes the inbound packet, then it's too big to forward. I had to track down the other side of this. FreeBSD used to resend the whole send queue if it got a too big message, even if the size did not change. Sending all at once made it pretty likely for the broken forwarder to get packets close enough to do LRO, which resulted in large enough packet sending to show up as network problems. I don't remember where the forwarder seemed to be, somewhere far away, IIRC.
	▲	cryptonector 8 months ago \| parent \| prev \| next [-]
		> PMTU just doesn't feel reliable to me because of poorly behaved boxes in the middle. The worst offender I've had to deal with was AWS Transit Gateway, which just doesn't bother sending ICMP too big messages. Passive PMTUD does NOT depend on ICMP messages.
	▲	immibis 8 months ago \| parent \| prev \| next [-]
		L2 not generating errors is expected behaviour - all ports on the L2 network are supposed to have the same MTU set
	▲	Hikikomori 8 months ago \| parent \| prev [-]
		They recently started supporting pmtud on tgw. But it wasn't a big deal really as it adjusted mss instead.

▲

JackSlateur 8 months ago | parent | prev | next [-]

path mtu discovery is worthless because the sending host does not control the path used

So it is not compatible with anycast, for instance, which is massively used everywhere

In the end, having no answer is better than having a most likely wrong answer

▲

mkj 8 months ago | parent | prev [-]

Would that help with UDP, or only TCP?

	▲	ajb 8 months ago \| parent \| next [-]
		That particular one, only TCP. There is a different one for UDP applications: https://www.rfc-editor.org/rfc/rfc8899 Because UDP is only a very thin layer, each layer on top (eg, QUIC) has to implement PLPMTUD; although, recently IETF standardised a way to extend UDP to have options and PLPTMUD is also specified for that: https://datatracker.ietf.org/doc/draft-ietf-tsvwg-udp-option...
	▲	8 months ago \| parent \| prev \| next [-]
		[deleted]
	▲	cryptonector 8 months ago \| parent \| prev [-]
		You can implement passive PMTUD with UDP if you like. It's more work for you, but it's perfectly doable.