Remix.run Logo
▲ ssl-3 a day ago

The phone company was amazingly reliable, at least in the States.

In all of my decades I recall only one telephone outage one time, at home. A truck (or something) had snagged a line feeding the house. It was repaired less than an hour after reporting it.

And I'm nostalgic for that kind of reliability, but it's gone now.

I was chatting with a tech who was installing a new county 911 system a few weeks ago. There was a time, not so long ago, when 911 had been robust: Fixed circuits to multiple different areas (to walk around network issues). Slow, redundant, robust analog modems on leased lines for caller data and call control. Eventually, things like PRI or T1.

Either way, there were great racks of redundant gear at the 911 PSAP for managing calls.

But now? It's just VOIP over the public Internet. 911 calls get the same not-special class of service that posting this comment does. The huge gear-filled racks are gone. In their place is a router and a switch, and a collection of dispatch PCs.

(There's a backup IP connection, of course. This backup connection uses cellular data.)

▲catlifeonmars a day ago | parent | next [-]

I was going to point out VoLTE as an example of a system which runs over IP but still retains a lot of the concepts from circuit switched days. In theory (if you’re a network operator), you could expose dedicated bearer provisioning with QoS guarantees for applications to build on top of. I guess my point is that you can have tighter QoS control if you can get off the public internet and go through carrier IPX instead.

▲g_p 17 hours ago | parent | next [-]

VoLTE and SIP do indeed retail a lot of the concepts of circuit switched. The challenge is that now you have the complexity of (seemingly invariably) a Telco managing a k8s stack on an OS they don't understand, and which has hundreds of CVEs per month they aren't patching. The failure modes of VoLTE/SIP are also a load more complex than CS domain issues - IMS interop issues and XML parsing delights await!

So I guess it's not only about getting off the internet - there is a real shift in complexity in these more modern stacks too.

▲catlifeonmars 10 hours ago | parent | next [-]

You are not wrong about the complexity of modern stacks.

However, VoLTE predates k8s adoption in most telcos and the physical network appliance == NF model is still alive and well, so it’s not necessarily a fair comparison. You’re also right about the added complexity of SIP/RTP (let’s agree not to talk about MMS lol).

▲wil421 12 hours ago | parent | prev [-]

That’s not true at all. Yes to the k8s but at least at my work we are scanned to death as soon as a new IP comes up on our network. Plus we have golden image updates at least every 30 days. We couldn’t even get SNMP traps out of our virtual firewalls because of “security”.

Containers are 10x better than monitoring and supporting some 30 years old switch in a central office that consumes more power than my cul de sac.

▲ssl-3 6 hours ago | parent | prev [-]

I think you just described how the PSTN used to run, except this way has more layers.

▲catlifeonmars 3 hours ago | parent [-]

Welcome to VoLTE :)

▲6510 a day ago | parent | prev | next [-]

Back in the days, every time I pealed open a bundle of those paper wrapped wires I had to stop and be amazed how amazingly reliable it was. EVERY TIME

https://www.facebook.com/groups/telecomarchaeology/posts/354...

This just doesn't look like something that will work wonderfully for 80+ years.

▲ssl-3 5 hours ago | parent [-]

That stuff is lead-sheathed, isn't it?

I haven't ever gotten into it myself, but I had a DSL connection that would go wonky on sunny evenings due to an increase in reception of a nearby AM radio transmitter that drowned out the useful signals. (Amusingly, the hardware on my end provided charts that pinned down the frequency perfectly.)

That circuit had lead sheathed cable for part of its path. After one call to retentions, I had an AT&T convention at my house once or twice per week.

They were all in uniforms, driving fancy logo'd trucks. They all failed to make it work correctly.

Until one day, some dude with facial piercings who was dressed like me in jeans and a black tshirt showed up. I don't know what back office they pulled this guy out of, but he insisted that this was his problem now, that he was the most qualified person to correct it, and that he would be working on it until it was identified and fixed, and that he would begin as soon as he went back to get a bucket truck.

I don't know what all he got into. He worked until well after dark. And then he came back to the door and pronounced with great confidence that he had found an issue with some old lead sheathed cable and corrected it, and that this would absolutely resolve my problem.

And it did. 10/10 stability after that.

(Shoulda kept his contact info. He'd have been a good one to hire.)

▲anon7000 17 hours ago | parent | prev [-]

I mean, going over the public internet gives you a pretty huge amount of redundancy. It’s common for individual companies to have outages, or one small neighborhood to have an issue, but has there ever been a global internet outage?

▲cowlevel 27 minutes ago | parent | next [-]

Only your neighbourhood needs to have an outage for your neighbourhood to have an outage. Nobody's saying it's okay if 911 goes down as long as it's still up for some people in the world.

▲manarth 15 hours ago | parent | prev | next [-]

Perhaps not completely global, but there have been some extremely widespread outages (usually from DNS or BGP misconfiguration).

- 2019: Verizon/BGP routing issue

- 2021: Verizon/FIOS routing issue

- 2021: Fastly CDN outage

- 2021: Comcast: San Francisco backbone outage

- 2021: Akamai: Edge DNS outage

- 2022: Rogers telecom routing issue

- 2023: Cloudflare: DNSSec expired certificates triggered an outage

- 2025: Cloudflare: core proxy outage

▲Angostura 16 hours ago | parent | prev | next [-]

It doesn’t give you what POTS used to give you - several days of battery backup that kept the phone system working in the event of power failure. I remember the huge racks of lead-acid batteries in the old exchange buildings

▲ssl-3 6 hours ago | parent [-]

The batteries were part of the trick, for sure. All of the telco gear ran on -48v, and was connected in parallel with the batteries.

In this way, grid power could come and go rapidly and the switches wouldn't even notice. There there was no handoff, or switchover time. The chemical batteries just changed from being held at a steady SoC to being discharged, which is something they can do without complaint or interruption. KISS.

After that, at least for the AT&T-built central office down the road from where I grew up, the generators would engage and keep things going indefinitely (or until fuel ceased to flow).

▲ButlerianJihad 6 hours ago | parent | prev [-]

There was that time when .com and .net were corrupted for 4 hours, and the death of Michael Jackson crashed Twitter, AIM, and Wikipedia

https://en.wikipedia.org/wiki/Internet_outage#List