HN meta: 99% packet loss on news.ycombinator.com

4 points by eqvinox a day ago | 5 comments

I'm currently seeing >99% packet loss to news.ycombinator.com (2606:7100:1:67::26):

  13. br06-te1-2.lwdc.americanis.net                   0.0%   231   14.3  17.2  14.0 172.5  19.5
  14. sdtc.sd-me01.te0-1.host1.20546.americanis.net   99.6%   231   12.9  12.9  12.9  12.9   0.0
  15. 2606:7100:1:67::26                              99.1%   231   11.8  11.8  11.8  11.8   0.0

Checked from both US and EU connections, same result, would be a very odd coincidence if it was a problem on my end. Trace looks like the uplink provider is having issues?

IPv4 seems to be unaffected.

▲ Bender a day ago | parent [-]

Packet loss over ICMP can be artificial based on the backplane load of routers, ACL's and artificial rate limits. They will de-prioritize responding to ICMP and fail to respond if CPU load is high but that does not mean the packets are not being forwarded 100%. Linux and BSD also have knobs for this behavior as well. HN is BSD.

Are you seeing TCP retransmits to port 443?

    for i in $(seq 69);do nc -vz -w1 news.ycombinator.com 443;sleep 2;done

Do that at the same time as watching

    watch -d -n4 --no-title 'nstat -a|grep -Ei "ret|los"'

It would be most useful to do this from your location and also from VM's on a few different providers in different locations to find which thing is not like the other. The 'nc' I am using is part of the nmap distribution. To force IPv6 replace the name with the IPv6 address.

▲

eqvinox a day ago | parent [-]

> Packet loss over ICMP can be artificial based on the backplane load of routers.

This doesn't generally apply to the end host; note the last hop in the trace I posted is the web server itself. Also, the second last hop having very similar loss% makes it likely that neither of the two systems is hitting rate limits (as they would be different).

> Are you seeing TCP retransmits to port 443?

If the webpage loaded normally, I wouldn't have investigated to begin with ;).

Right now, the loss on TCP SYNs is about 70%. (mtr -T -P 443) Second to last hop still has roughly the same loss. Something's h0rked at americanis.net.

▲

Bender a day ago | parent [-]

If you are confident where the loss is then the only thing you could really do within HN is email Daniel (dang) and ask him to replicate the issue and open a ticket with their upstream provider to check routes and for losses. hn@ycombinator.com

Outside of HN I suppose you could check the Nanog mailing lists [1] and chat servers to see if there is any discussion ongoing for those networks.

[1] - https://lists.nanog.org/mailman3/

▲

eqvinox 9 hours ago | parent [-]

I decided to wait a bit, and it's fixed now by way of "americanis.net" no longer being on the path.

	▲	Bender 7 hours ago \| parent [-]
		Groovy. Glad it worked out.