| ▲ | krystofbe 5 hours ago |
| Looks like a DNSSEC issue, not a nameserver outage. Validating resolvers SERVFAIL on every .de name with EDE: RRSIG with malformed signature found for
a0d5d1p51kijsevll74k523htmq406bk.de/nsec3 (keytag=33834)
dig +cd amazon.de @8.8.8.8 works, dig amazon.de @a.nic.de works. Zone data is intact, DENIC just published an RRSIG over an NSEC3 record that doesn't validate against ZSK 33834. Every validating resolver therefore refuses to answer. Intermittency fits anycast: some [a-n].nic.de instances still serve the previous (good) signatures, so retries occasionally land on a healthy auth. Per DENIC's FAQ the .de ZSK rotates every 5 weeks via pre-publish, so this smells like a botched rollover. |
|
| ▲ | qazwsxedchac 4 hours ago | parent | next [-] |
| So a single configuration mistake in a single place wiped out external reachability of a major economy. It happened in the evening local time and should be fixable, modulo cache TTLs, by morning. This will limit the blast radius somewhat. Still, at this level, brittle infrastructure is a political risk. The internet's famous "routing around damage" isn't quite working here. Should make for an interesting post mortem. |
| |
| ▲ | gerdesj an hour ago | parent | next [-] | | "The internet's famous "routing around damage" isn't quite working here." DNS is a look up service that runs on the internet. Internet routing of IP packets is what the internet does and that is working fine (for a given value of fine). You remind me of someone using the term "the internet is down" that really means: "I've forgotten my wifi password". | | | |
| ▲ | belorn 2 hours ago | parent | prev | next [-] | | I am reminded of the warning that zonemaster gives about putting your domain name servers on a single AS, as is common practice for many larger providers. A lot of people do not want others to see this as a problem since a single AS is a convenient configuration for routing, but it has the downside of being a single point of failure. Building redundant infrastructure that can withstand BGP and DNS configuration mistakes are not that simple but it can be done. | | |
| ▲ | walrus01 31 minutes ago | parent [-] | | As the CPU/RAM resources to run an authoritative-only slave nameserver for a few domains are extremely minimal (mine run at a unix load of 0.01), it's a very wise idea to put your ns3 or something at a totally different service provider on another continent. It costs less than a cup of coffee per month. |
| |
| ▲ | pocksuppet 3 hours ago | parent | prev | next [-] | | DNS is a centralization risk, yes. Somehow we've decided this is fine. DNSSEC isn't the only issue - your TLD's nameservers could also be offline, or censored in your country. | | |
| ▲ | skywhopper 3 hours ago | parent | next [-] | | DNS is barely centralized. Is there an alternative global name lookup system that is less centralized without even worse downsides? | | |
| ▲ | pocksuppet 2 hours ago | parent [-] | | BGP, but the names in question are limited to 128 bits, of which at most 48 will be looked up, and you don't get to choose which 48 bits are assigned to you. |
| |
| ▲ | greatgib 2 hours ago | parent | prev | next [-] | | Normally it should not have been, with cache and all, but that was the past... Think about what would happen the day that letsencrypt is borken for whatever reason technical or like having a retarded US leader and being located in the wrong country. Taken into account the push of letsencrypt with major web browsers to restrict certificate validities for short periods like only a few days... | | |
| ▲ | muvlon 2 hours ago | parent [-] | | Let's Encrypt has to be down for days before people begin to feel the pain. DNS is very different, it breaks stuff immediately everywhere. | | |
| ▲ | tharkun__ an hour ago | parent [-] | | No it doesn't. DNS breaks as soon as TTLs run out. It's your choice to set them so low that stuff breaks immediately. |
|
| |
| ▲ | cyberax 3 hours ago | parent | prev [-] | | Not really? .com and .net are still up If Let's Encrypt goes down, half of the Internet will become inaccessible in a week. | | |
| |
| ▲ | Muromec 3 hours ago | parent | prev | next [-] | | >So a single configuration mistake in a single place wiped out external reachability of a major economy. And fuck nothing at all happened as a result. | | | |
| ▲ | lschueller 3 hours ago | parent | prev | next [-] | | I have a bad feeling, that the impact will be quite severe for some services, as monitoring, performance, and security services might get disrupted. and just cleaning up is a big mess.. Worst case, some ot will experience outage and / or damage. But maybe I am just overestimating the severity of this. | |
| ▲ | walrus01 4 hours ago | parent | prev | next [-] | | It looks like a failed key replacement during a scheduled maintenance event. Normally this sort of thing is thoroughly tested and has multiple eyes on for detailed review and planning before changes get committed, but obviously something got missed. | |
| ▲ | the8472 3 hours ago | parent | prev [-] | | fail-closed protocols have introduced some brittleness. A HTTP 1.0 server from 1999 probably still can service visitors today. A HTTPS/TLS 1.0 server from the same year wouldn't. |
|
|
| ▲ | dlopes7 4 hours ago | parent | prev [-] |
| I love how I work with IT for 20 years and don't understand a single acronym here other than DNSSEC |
| |
| ▲ | icedchai 3 hours ago | parent | next [-] | | I've been in IT 30+ years, been running DNS, web servers, etc. since at least 1994. I haven't bothered with DNSSEC due to perceived operational complexity. The penalty for a screw up, a total outage, just doesn't seem worth the security it provides. | | |
| ▲ | gerdesj 29 minutes ago | parent | next [-] | | That was my experience too until I decided that just running email systems for 30 odd years when HN says that is unnatural piqued my weird or something! I ran up three new VMs on three different sites. I linked all three systems via a private Wireguard mesh. MariaDB on each VM bound to the wg IP and stock replication from the "primary". PowerDNS runs across that lot. One of the VMs is not available from the internet and has no identity within the DNS. The idea is that if the Eye of Sauron bears down on me, I can bring another DNS server online quite quickly and fiddle the records to bring it online. It also serves as a third authority for replication. I also deployed https://github.com/PowerDNS-Admin/PowerDNS-Admin which is getting on a bit and will be replaced eventually but works beautifully. Now I have DNS with DNSSEC and dynamic DNS and all the rest. This is how you start signing a zone and PowerDNS will look after everything else: # pdnsutil secure-zone example.co.uk
# pdnsutil zone set-nsec3 example.co.uk
# pdnsutil zone rectify example.co.uk
Grab a test zone and work it all out first, it will cost you not a lot and then go for "production".My home systems are DNSSEC signed. | |
| ▲ | qingcharles 36 minutes ago | parent | prev [-] | | How simple sysadmin was in 1994 with no cryptography on any protocol. Everything could be easily MITM'd. Your credit card number would get jacked left and right in the 90s. | | |
| ▲ | gerdesj 25 minutes ago | parent [-] | | Cool. Feel free to explain how to tighten things up. I've just given them part of a recipe for using DNSSEC. I suspect you are not actually human .. qingcharles. |
|
| |
| ▲ | walrus01 4 hours ago | parent | prev | next [-] | | To be fair, advanced real world knowledge of public/private key PKIs (x.509 or other), things like root CAs, are a fairly esoteric and very specialized field of study. There's people whose regular day jobs are nothing but doing stuff with PKI infrastructure and their depth of knowledge on many other non-PKI subjects is probably surface level only. | | |
| ▲ | hannob 4 hours ago | parent | next [-] | | I know quite a bit about PKI and X.509, and I can tell you that much: the overlap with how DNSSEC works is limited. | | |
| ▲ | silisili 3 hours ago | parent [-] | | As is the overlap between DNSSEC and DNS itself, to be honest. I once worked at the level of administering DNSSEC for 300+ TLDs. It's its own world. When that company was winding down, I tried to continue in the field but the most common response (outside of no response, of course), was 'we already have a DNS team/vendor/guy.'
And well, then things like this happen. I won't throw stones though, it's a lot to learn and can be incredibly brittle. |
| |
| ▲ | hathawsh 4 hours ago | parent | prev | next [-] | | Is that actually true, though? Even though it's not really my job, I find myself debugging certificates and keys at least once a month, and that's after automating as much as possible with certbot and cloud certificates. PKI always seems to demand attention. | | |
| ▲ | walrus01 4 hours ago | parent [-] | | In my initial comment, I meant more in terms of complexity and planning from the perspective of the people who are running the public/private key infrastructure on the other side/upstream of what you're doing as a letsencrypt end user. Broadly similar general concept to the team responsible for the DNSSSEC signing keys for an entire ccTLD. Yeah a x509 PKI / root CA is a very different thing than DNSSSEC but they have a number of general logical similarities in that the chain of trust ultimately comes down to a "do not fuck this up" single point of failure. |
| |
| ▲ | mschuster91 4 hours ago | parent | prev [-] | | It's not made easier by the fact that a lot of cryptography is either very old and arcane or it's one hell of a mess of code that doesn't make sense without reading standards. I had the misfortune of having to dig deep into constructing ASN.1 payloads by hand [1] because that's the only thing Java speaks, and oh holy hell is this A MESS because OF COURSE there's two ways to encode a bunch of bytes (BIT STRING vs OCTET STRING) and encoding ed25519 keys uses BOTH [2]. And ed25519 is a mess in itself. The more-or-less standard implementation by orlp [3] is almost completely lacking any comments explaining what is going on where and reading the relevant RFCs alone doesn't help, it's probably only understandable by reading a 500 pages math paper. It's almost as if cryptographers have zero interest in interested random people to join the field. End of rant. [1] https://github.com/msmuenchen/meshcore-packets-java/blob/mai... [2] https://datatracker.ietf.org/doc/html/rfc8410#appendix-A [3] https://github.com/orlp/ed25519/tree/master | | |
| ▲ | Muromec 2 hours ago | parent | next [-] | | The trick to asn.1 is to generate both parser and serializer from the spec. Elliptic curve math on the other hand is ... yeah, you need to know the math and also know the tricks to code that implements it. Both of those have steep learning curve, but it's hardly because it's a mess or it's old. | | |
| ▲ | tptacek 2 hours ago | parent | next [-] | | The trick to ASN.1 is to serialize/unserialize it backwards. | | | |
| ▲ | mschuster91 2 hours ago | parent | prev [-] | | > Both of those have steep learning curve, but it's hardly because it's a mess or it's old. Bitpacking structures used to be important in the 60s. That time has passed, unless you're dealing with LoRa, NFC or other cases of highly constrained bandwidth there are way better options to serialize and deserialize information. It's time to move on, and the complexity of all the legacy garbage in crypto has been the case of many a security vulnerability in the past. As for the code, it might be personal preference but I'd love to have at least some comments referring back to a specification or original research paper in the code. | | |
| ▲ | Muromec 2 hours ago | parent | next [-] | | I think you misunderstand the problem asn.1 solves and constrains it works within (both 30 years ago and now). We sure can have a better one now once we learned all the lessons and know what good parts to keep, but this critique of bitpacking is misplaced. | |
| ▲ | Avamander 2 hours ago | parent | prev [-] | | ASN.1 is not used because of just bitpacking. There are other benefits to ASN.1 and it's probably one of the least problematic parts there. People who have thought they can do better have made things like PGP. It's one of the worst cryptographic solutions out there. You're free to try as well though. | | |
| ▲ | Muromec 2 hours ago | parent [-] | | People who though they can do better did JWT, that is not complicated at all and has no bugs as well. Also solves 20% of what asn.1 is used for. |
|
|
| |
| ▲ | tptacek 3 hours ago | parent | prev | next [-] | | The typical vector for entering cryptography as a professional is called "grad school". | |
| ▲ | cyberax 3 hours ago | parent | prev [-] | | X.509 is a deep legacy, but at least at this point it's well tested. > because that's the only thing Java speaks No, it most definitely is not. You can just construct a private key directly in BouncyCastle: https://downloads.bouncycastle.org/java/docs/bcprov-jdk18on-... I'm 100% certain that you also can do that with raw java.security. I did that about 15 years ago with raw RSA/EC keys. You can just directly specify the private exponent for RSA (as a bigint!) or the curve point for EC. Ditto for ed25519, you can just take the canonical implementation from DJB. And you really really shouldn't do that anyway, please just use OpenSSL or another similar major crypto library. | | |
| ▲ | Muromec 2 hours ago | parent | next [-] | | I wouldn't recommend touching openssl (the library, command line tools are okay-ish) with anything that breaths life. | |
| ▲ | mschuster91 2 hours ago | parent | prev [-] | | > I'm 100% certain that you also can do that with raw java.security. I tried that, the problem is Meshcore specific - they do their own weird shit with private and public keys [1]. Haven't figured out how to do the private key import either, because in the C source code (or in python re-implementations) Meshcore just calls directly into the raw ed25519 library to do their custom math... it's a mess. [1] https://jacksbrain.com/2026/01/a-hitchhiker-s-guide-to-meshc... |
|
|
| |
| ▲ | bflesch 3 hours ago | parent | prev [-] | | Don't worry, that's by design ;) |
|