Remix.run Logo
film42 6 days ago

Zoom CEO: Hi, we'd like an SLA credit for the global outage you caused our company.

GoDaddy: I am so sorry about that. I can offer you a one-time coupon for $10 off your next purchase or renewal. Would you like me to apply this to your account?

---

Most companies just hope an apologetic zoom call is enough to retain your business, and most of the time it works. Not enough has been written about the asymmetry of your SLA credits to your revenue impact for a given vendor outage and how that should guide your build vs buy decision framework.

mikeocool 6 days ago | parent | next [-]

You probably don’t want to optimize for the SLA credit making up for a significant part of your lost revenue — because that would mean when things are operating normally, you don’t have much of a profit margin.

SLA’s are generally more helpful for getting out of long term contracts with unreliable vendors than actually making up for revenue lost during an outage.

kevincox 6 days ago | parent [-]

SLA credits are an incentive for the service provider not making up for lost revenue from the outage.

If you have 100% SLA credit under 99% availability you can't aford to be less than 99% available and I know that your SLA means something to you, not just an aspirational bullet point.

Geezus_42 6 days ago | parent | prev | next [-]

Why would you use godaddy for a service as large as Zoom? They have been garbage for years. The way they locked out their ACME api for anyone but top tear clients sealed the deal for me. I would never trust them.

signal11 6 days ago | parent | next [-]

From the linked article

> This block was the result of a communication error between Zoom’s domain registrar, Markmonitor, and GoDaddy Registry, which resulted in GoDaddy Registry mistakenly shutting down zoom.us domain.

Markmonitor is used by some fairly large corps and web properties. It’ll be interesting to find out exactly what this miscommunication was.

6 days ago | parent [-]
[deleted]
0x0000000 6 days ago | parent | prev | next [-]

They don't use Godaddy directly. Godaddy is the registry for .us. Zoom's registrar is MarkMonitor, who appear to be at fault for this outage.

subscribed 6 days ago | parent | next [-]

No, Mark Monitor have requested the correct change (EPP status code `ServerUpdateProhibited), GoDaddy messed it up.

(I'm not affiliated with either, but happen to know the technical details of the outage)

SahAssar 6 days ago | parent [-]

Could you let us know how you know the technical details? Is there some public info the rest of us haven't been clued into?

sgarland 6 days ago | parent | prev [-]

Never heard of MarkMonitor before. Not a great start.

I had Google Domains for years, until they abruptly and bizarrely abandoned it, then I left for Porkbun. Never had a problem with either of them. I get yearly auto-renewal notices. Everything works, and it’s very boring, which is precisely what I want from a registrar.

Moto7451 6 days ago | parent | next [-]

I worked at a company that used MM and was involved in some of the domain work.

One of the really nice things about the service is they handle a lot of the general business continuity and security stuff that can really suck with traditional registrars. One of their main lines of work is they’ll work with you to resolve tld-squatting and typo-squatting by working directly with the registrars.

Even before an infinite number of vanity or scammy tlds started showing up it would be pretty difficult to find <your-growing-unicorn-startup>.biz to add to your portfolio of domains since the owner may just have forgotten to update their email in their registrar and were coasting on a 10 year registration. Maybe the squat was intentional and it’s now a 1:1 replica of your homepage with a phishing or other credit card scam going. Stuff like that really sucks to do yourself while handling your other responsibilities. MM was pretty successful at getting in touch with the owners in the first place and having the registrar yank and transfer in the latter case. YMMV of course.

Once a lot of tlds started showing up, and especially the porn related ones, they worked with the new registrars directly (like GoDaddy in the .us case here) in the “sunrise period” to make sure something like google.xxx doesn’t become a front page article about an actual porn site (in case you’re wondering, that one doesn’t go anywhere at all). Your other options are to work directly with each registrar or ICANN.

throwanem 6 days ago | parent | next [-]

Oh, I didn't know they'd been around since '99. They called me on behalf of a Hollywood titan about one of my hobby domains, which happened to partially coincide at a dictionary word with one of their client's trademarks, some time around 2006. I don't recall that the approximate paralegal I spoke with actually identified the company; I never forgot the call, but hadn't thought to check who manages the studio's own domain. Go figure.

I found them surprisingly easy to deal with, and happy to have me on record that my toy domain had nothing to do with either their client or any money. I assumed as long as that remained the case I would never hear from them again, and for the decade or so longer I kept the domain, that was exactly how things went.

prmoustache 6 days ago | parent [-]

I may be wrong but I think I saw MarkMonitor changing hands a year or two ago so the MarkMonitor of today might not be at the same level and quality of service as before.

throwanem 6 days ago | parent [-]

Yeah, in 2022 acquired by some kind of sketchy rollup of lots of legacy/web-1.0 firms or what remained of them, it looks like.

Oh, well. It's been a long time since I was so naïve as not to do a quick informal trademark/brand search before I register a new domain, so I don't really expect to hear from them again any time soon, either.

sidewndr46 6 days ago | parent | prev [-]

I'd argue the ability of a private company to exert control over all TLDs on the behalf of their clients is indicative of a problem in the domain registrar system. Not a "service"

dewey 6 days ago | parent | prev | next [-]

That’s because you are maybe not in the market for MarkMonitor. If you check the whois for any global brands chances are they are held by MarkMonitor. Just like you don’t use EY as your tax advisor.

sgarland 6 days ago | parent [-]

Genuinely, I don’t understand how anything other than uptime matters for a domain registrar.

What services are they offering that makes them attractive to corporations?

slyall 6 days ago | parent | next [-]

The are supposed to also filter things like complaints. If somebody complains I'm sending spam and I only pay $20/year then my registrar might lock my domain and then I have to work to get it back online.

Mark Monitor will apply a lot more filtering to complaints.

Ironically this is allegedly what happened in this case, a complaint about the domain got it taken offline.

reilly3000 6 days ago | parent | prev | next [-]

They generally do full service brand monitoring to protect IP and maintain continuity. You would outsource monitoring for trademark infringement to them, and be certain that domain renewals are done perfectly for a portfolio of high value domains.

lolinder 6 days ago | parent [-]

Which is why this outage is so weird: the entire point of paying MarkMonitor is ensuring that absolutely nothing goes wrong with a very fraught process, and they seem to have just taken down one of the biggest brands they support.

throwanem 6 days ago | parent [-]

Precisely. You pay a company like this the nosebleed-inducing fees they charge so that this exact event never happens. That assurance, and not the mechanics of domain registration or canned web searches or whatever else, is their product.

It's like, as I'm sure I'm paraphrasing from something I read God alone knows how many years ago, if your publicist lets you walk into a press event with a giant blob of snot hanging out of your nose. There surely is a reason why that error occurred, and it probably is at least a pretty good reason. But no one is very surprised to see the intro invite from your new publicist.

It isn't a relationship you blow up on a whim, but Zoom that can't route call traffic is Zoom that's not generating revenue, and while the reputational impact is negligible if it happens once, it had really better happen only once. Zoom is the incumbent; no one remembers they were revolutionary once, now everyone only notices the parts they don't like. (Being a skilled but politically naïve sysadmin is much the same.)

Basically, this is why Ma Bell - which had about the only stronger possible "uptime" expectation, in that no one uses Zoom for 911 - was so uptight you couldn't even plug in a modem until about five minutes before divestiture, and specified everything down to the number of turns in the splices their technicians made. There was a fad among programmers, when I was a child, to consider such practices stodgy.

macintux 6 days ago | parent [-]

Obligatory Indiana Bell building rotation link.

https://www.archdaily.com/973183/the-building-that-moved-how...

throwanem 6 days ago | parent [-]

I did not know that story!

What a paragon of engineering.

toast0 6 days ago | parent | prev | next [-]

Like others said, uptime for a registrar barely matters. For an important domain, I don't want anything to change, and if the registrar is down, nothing will change, so that's good.

What MarkMonitor can provide is things like facilitating RegistryLock, which makes it even harder for changes to be made. And account reps that know what's going on. I hate working with account reps, but if they're knowledgable and easy to work with, it's ok.

They do some trademark monitoring (thus the name), if you want to get your own related app taken down from Google Play :p (I'm not bitter, it was amusing). And presence services if you need to hold a domain in a weird location that wants a presence, they can probably arrange it, which is handy at times.

I'd love to know more details on this incident, MarkMonitor had a bulletproof reputation as a registrar that won't fuck up. Godaddy doesn't, but then I didn't realize they had taken over the contract for .us

kryptiskt 6 days ago | parent | prev | next [-]

They can offer humans in the loop, and those cost a lot. Like, a real live human will contact you and ask if you really want to transfer microsoft.com to Shady Shell Company (Bermuda) Ltd. Porkbun's pricing model is less attractive when your domains are worth billions to you.

BrandoElFollito 6 days ago | parent | prev [-]

Why would uptime matter that much for a registrar?

(As opposed to a DNS server, including root servers - and even then DNS has provisions for downtime, not to mention redundancy in configurations)

the-rc 6 days ago | parent | prev [-]

MarkMonitor has been around forever. It's used by many of the largest companies. I remember quite a few Google outages that could be tracked down to MM issues.

Geezus_42 6 days ago | parent | prev | next [-]

I just remembered, they also can't do DKIM correctly. What good is a DNS provider that can't follow standards?

technion 6 days ago | parent | prev [-]

Companies as big as zoom are still perfectly capable of having a high level VIP decide "we're going to use GoDaddy because I saw their Superbowl ad".

pavelstoev 6 days ago | parent | prev | next [-]

Can’t have an apologetic zoom call when zoom is down …

crazygringo 6 days ago | parent | prev | next [-]

If there were symmetry, then renewing the domain would cost millions instead of $20 or whatever it is, to cover the payouts. Is that what you want?

If it is, you can buy custom insurance for the event from an insurance company, and pay the same kind of yearly fee.

And remember that with build vs buy, what you build will often be worse than what you buy, because at least what you buy is getting bugs fixed from bug reports across the world from other customers. An internal tool will rarely be as stress-tested and battle-hardened as what you can buy.

chazeon 6 days ago | parent | prev | next [-]

I remember crowdstrike outage offers starbucks coupons? that’s way to go.

6 days ago | parent | prev [-]
[deleted]