So a single configuration mistake in a single place wiped out external reachability of a major economy. It happened in the evening local time and should be fixable, modulo cache TTLs, by morning. This will limit the blast radius somewhat.

Still, at this level, brittle infrastructure is a political risk. The internet's famous "routing around damage" isn't quite working here. Should make for an interesting post mortem.

I love how I work with IT for 20 years and don't understand a single acronym here other than DNSSEC

A real party killer if I have ever seen one.

Interesting "bus problem" to have in a scenario where everyone who is qualified, experienced and trusted enough to commit lives changes (or perform a revert, undo results of a botched maintenance, etc) in an emergency situation is not completely sober.

> which now breaks because the central organisation managing this certificate has an outage

The ".de" TLD is inherently managed by a single organization, and things wouldn't be much better if its nameservers went down. Some of the records would be cached by downstream resolvers, but not all of them, and not for very long.

> we took the decentralized platform DNS was and added a single-point-of-failure certificate layer on top of it

DNSSEC actually makes DNS more decentralized: without DNSSEC, the only way to guarantee a trustworthy response is to directly ask the authoritative nameservers. But with DNSSEC, you can query third-party caching resolvers and still be able to trust the response because only a legitimate answer will have a valid signature.

Similarly, without DNSSEC, a domain owner needs to absolutely trust its authoritative nameservers, since they can trivially forge trusted results. But with DNSSEC, you don't need to trust your authoritative nameservers nearly as much [0], meaning that you can safely host some of them with third-parties.

[0]: https://news.ycombinator.com/item?id=47409728

DNSSEC doesn't change the degree to which DNS is decentralized. It's always been hierarchical. In the absence of caching, every DNS query starts with a request to the root DNS servers. For foo.com or foo.de, you first need to query the root servers to determine the nameservers responsible for .com and .de. Then you contact the .com or .de servers to ask for the foo.com and foo.de nameservers. All DNSSEC does is add signatures to these responses, and adds public keys so you can authenticate responses the next level down.

A list of root nameserver IP addresses is included with every local recursive DNS resolver. The list changes, albeit slowly, over the years. With DNSSEC, this list also includes public keys of those root servers, which also rotate, slowly.

What you see here is decentralisation working. The issue is with the operator of the de TLD, and as such only that TLD is affected. DNS is not decentralised in such a way, that multiple organisations run the infrastructure of a TLD, those are always run by a single entity.(.com and .net are operated by Verisign)

So what the issue is, that the operator has, does not change the impact.

Welp. I think can call it on DNSSEC now.

If it turns out the DNSSEC issue was caused by threat actors, this downstream effect could very well have been the reason to do it.

Aged like a milk.

I remember when .com went down, in July 1997.

https://archive.nytimes.com/www.nytimes.com/library/cyber/we...

DENIC apparently resolved all .de domains to NXDOMAIN in 2010: https://www.theregister.com/2010/05/12/germany_top_level_dom...

There's a good index of major DNSSEC outages here, https://ianix.com/pub/dnssec-outages.html

It's Germany, pessimistic time estimation + 1/3 and you are in a realistic time frame for the issue being resolved.

Well it was already very late in the day (21-22?) so the impact was not big I would say

Germany isn't as big as you think.

https://i.imgur.com/eAwdKEC.png

Edit: Alternative link: https://www.cyberciti.biz/media/new/cms/2017/04/dns.jpg

We should frame it as "all .de domains are ready to be impersonated because everyone will disable DNSSEC".

What would I need to rant about? Sometimes the world does my ranting for me.

Maybe he drank a little too much Malört with the DENIC team last night?

He’s busy with MathAcademy earning XP-SEC

doesn't this event speak for itself though?

Perhaps he's moribund

But we're Germans, and we need someone to blame.

Even when every site in the world’s 3rd biggest economy goes down it’s still just a ‘Partial’ service disruption :D

Whole Germany is offline. DENIC: "Partial Service Disruption". That's one way to phrase it.

At least they have some humor left.

Edit: Now even the humor is gone.

It says "Server Not Found" now

"All Systems Operational"

I don't even enable DNSSEC in Unbound. There just isn't enough adoption yet for me to feel like I am missing out on something, yet.

"Cloudflare Radar data shows 8.11% of domains are signed with DNSSEC, but only 0.47% of queries are validated end-to-end." [1]

Zones I may care about:

- Amazon.com: unsigned

- My banks: unsigned

- Hacker News: unsigned

- Email that I do not host: unsigned

- My power companies billing: unsigned

- I found some! id.me and irs.gov are signed.

[1] - https://technologychecker.io/blog/dnssec-adoption

Just before the outage happened I updated multiple client servers. That was a very stressfull hour trying to figure out why nothing works.

Same haha

SAMEEEEE !!!

Internetfreie Dienstage, 21st century variant of Autofreie Sonntage.

Using this newfangled thingamabob on a silent holiday will result in the police kicking in your door the next morning.

Danke Merkel

Ok children, sit down and listen, uncle Culonavirus will tell you a story:

"It all began with the decommissioning of the last nuclear power plant, ..."

We shall transmit the postmortem to you via fax within 25 business days, ja.

The irony is that DNS is a global and distributed system meant to be resilient. It’s the DNSSEC layer on top in this case causing problems.

Both examples open for me

amazon.de, spiegel.de are down for me, too. heise.de works, but that might've been cached somewhere on my side.

idealo.de, ebay.de, and spiegel.de are down, but amazon.de opens for me.

Common paint point with DNSSEC. It’s brutal in the domain industry because when you buy a name with DNSSEC enabled it oftentimes can’t be setup to resolve due to these sorts of issues. Typically seller needs to deactivate first.

If so, it still worked for several hours after the maintenance was completed.

This works:

    $ unbound-host -t A www.denic.de
    www.denic.de has address 81.91.170.12

    $ unbound-host -D -t A www.denic.de
    www.denic.de has address 81.91.170.12
    validation failure <www.denic.de. A IN>: signature crypto failed from 194.246.96.1 for DS denic.de. while building chain of trust

EDIT My explanation was wrong, this is not how keytags work. The published keytag data is consistent:

    de. 3600 IN DNSKEY 256 3 8 AwEAAfRLmzuIXVf7x5A0+U7hke0dS+GEJG0EdPhnOthCCLhy0t0WqLyoXJOhnfsTJ8vQX5fd9qOJc9gyr3SWJZkXAhPm3yPSC7FWWHF70WZTKKM9CekmKdqwMwq6ZCjMSUcecCuSF4Sbt1MRszV7rFmfGVklA1l5UzNbqwD+Dr5vfcLn ;{id = 33834 (zsk), size = 1024b}
    de. 3600 IN DNSKEY 257 3 8 AwEAAbWUSd/QN9Ae543xzdiacY6qbjwtZ21QfmdgxRdm4Z7bjjHWy249uqxCyjjjoS4LDoRDKmj7ElffMKvTWKE1qFKu0p8TUy4wyhX0M+m5FUjvQ3CiZMi+qY7GSHA5B+Zd73cidmnTeb3e8lso6jEsXg05/VZ2AyAqWF6FexEIFxIqiwwLk4UP0BwZ17Ur3q1qx9VSbPMyHgQ9d6nHUN1EEJsTDA2v0vKumsUyp74ZanRZ/bB/6IzpaaZyr5BLF5pSCNdbRNjVmkwYD0993vm79LueyOeibsoHRc16jhALrIJou1PFjdq7YQsYN0KtqRiJtaAfPprDBREpeamPuW/MnW0= ;{id = 26755 (ksk), size = 2048b}
    de. 3600 IN DNSKEY 256 3 8 AwEAAbTe1PJi8EgIudNGb+KRTxBL2aCu5rXkZ+aIe/TC88pwRdrXYeXODp1ihZWFop5CrbWRBLrk/YUPBE8aBc6oJP+58dSkdMLYkjSkmvdvYx+zXnRLWlF2bapxvZxshATJDfGjGbCiWxKEOoyRx3UhICtHC+cUSddsEvzfacUcBb6n ;{id = 32911 (zsk), size = 1024b}
    de. 3600 IN RRSIG DNSKEY 8 1 3600 20260519030655 20260505013655 26755 de. ke56T5GZt/X6zMBAF+ouyCTnAd7RY7MsnDcfa9jyyOwSouRXhvzim/V13JDTMBAnpAHxWQXoruXrAZ6A6re5N+8Pp2utVkAEKTWs0r4UOLNKoZ2+zMwNplKjNNnY5PJIbHfa5myyziLiIsi//qDIgQEACFk+pZcHXrRdqRoXPCL3UtfaXjk3+duDQdlPnYsJys5UshjVpkALSMChW7J0anzr0sG+f9ytstBneymMwFYOUC3NqbejbLPZsXGPZBQKPAoVJuV5q3znopbcqrDFfjI7bmX3QPYNvOaiT1ElBfi2piJVpDzMaMAmm2jCmvrf5VeTOBccMroh8sBtDPsaEg== ;{id = 26755}

    de. 86400 IN SOA f.nic.de. dns-operations.denic.de. 1778014672 7200 7200 3600000 7200
    de. 86400 IN RRSIG SOA 8 1 86400 20260519205754 20260505192754 33834 de. aZoiAJ+PaHUDVSHNXfV/R26ZK3GpFB7ek2Z46VnZdmPEDaTww+a7PkiQ98W83xohUunXYSvQCMeGYfUre5UT76eBKThdxW2a6ImX9/x/oEzQ9x/69Y/NSeTckOv9m3HCLBOug01op1koiHOIAVEvonOmXEHHqo1P4sR/fNbcVg4= ;{id = 33834}

not all: https://www.heise.de/ works

cache

Seems to be up now?

May 5, 2026 23:28 CEST

May 5, 2026 21:28 UTC

INVESTIGATING

Frankfurt am Main, 5 May 2026 – DENIC eG is currently experiencing a disruption in its DNS service for .de domains. As a result, all DNSSEC-signed .de domains are currently affected in their reachability. The root cause of the disruption has not yet been fully identified. DENIC’s technical teams are working intensively on analysis and on restoring stable operations as quickly as possible. Based on current information, users and operators of .de domains may experience impairments in domain resolution. Further updates will be provided as soon as reliable findings on the cause and recovery are available. DENIC asks all affected parties for their understanding. For further enquiries, DENIC can be contacted via the usual channels.

They did now! https://x.com/denic_de/status/2051779175908774148

its gonna be all .de domains once caches dry out, anything that still works right now is bound to eventually fail until the underlying issue is resolved

Amazon is completely down in Germany. Not only on amazon.de, even in the app.

maybe your upstream doesn't validate DNSSEC?

cache

Security is not just a solution to availability. It is also to keep sensitive data (PII, or business secrets, or passwords, or cryptographic private keys, and so on) away from the hands of bad actors.

If I’m unable to use Amazon for 24 hours it doesn’t really matter. If a photo copy of my passport is leaked that’s worries and potential troubles for years.

Security = Confidentiality + Integrity + Availability

or alternatively,

Security = (exclude unauth'd reads) + (exclude unauth'd writes) + (include auth'd reads and auth'd writes)

Gotta satisfy all parts in order to have security.

Solar Flares

bmw.de is down for me too