But they'd have to be on the same network as me to do that attack, right?

Yep, like ECHELON and friends are. The metadata recorded about your (all of our) traffic is probably enough to perform the timing attack.

	▲	shadowgovt 4 hours ago \| parent [-]
		Hey, if ECHELON snuck a listener into my house, where six devices hang out on a local router... Good for them, they're welcome to my TODO lists and vast collection of public-domain 1950s informational videos. (I wouldn't recommend switching the option off for anything that could transit the Internet or be on a LAN with untrusted devices. I am one of those old sods who doesn't believe in the max-paranoia setting for things like "my own house," especially since if I dial that knob all the way up the point is moot; they've already compromised every individual device at the max-knob setting, so a timing attack on my SSH packet speed is a waste of effort).