Remix.run Logo
▲ throw0101a 9 hours ago

In August an attacker was able to get a certificate issued even with multi-perspective validation:

> For 33 hours in late August, a rival network hijacked a block of Hetzner-hosted infrastructure serving Softaculous and Virtualizor, using it to issue a valid TLS certificate and push a malicious software update. The stolen route was RPKI-invalid the entire time, on file, checkable by anyone. It still won on most of the internet.

> At 20:57 UTC on 28 August, a network nobody at Hetzner or Softaculous had dealt with before started announcing 162.55.80.0/24 to the internet. It was a small, unremarkable-looking slice of Hetzner’s own address space — the block hosting Softaculous’s client area, its billing pages, and the update endpoint that every Virtualizor-licensed control panel on the internet periodically calls home to. For the next 33 hours, with an 11-hour lull in the middle, a meaningful share of global traffic bound for that block went to the hijacker instead. Some of it was Let’s Encrypt’s own domain-control validation, which is how the attacker walked away with a certificate for virtualizor.com that no browser would ever flag.

* https://cloudpunk.blog/post/bgp-hijack-rpki-gap/

So it's all very well to have multiple network locations to prevent hijacking on the path (i.e., you have to attack multiple paths), but if the network as a whole thinks the destination is someplace else then all those paths will lead to the same (false) location.

Perhaps the CAB should consider RPKI validation as a "MUST" rather than piling on multiple sources of validation who still have the same single point of failure (the spoofing of the destination).

▲toast0 7 hours ago | parent | next [-]

The hijacker effectively established control over the domain. That's a properly issued cert in my book. There's no part of a domain control cert issuance where the CA is expected to establish the identity of the domain owner or obtain their consent.

▲organsnyder 6 hours ago | parent [-]

That's not how it used to be, though. It used to be that you had to prove a much higher level of ownership, even for standard, non-"enhanced" certs. There was one time we discovered that the business hadn't had its articles of incorporation renewed in the state, and the validation process by the cert issuer caught it.

I do prefer the current model (the trade-offs are worth it, IMHO), but it wasn't always this way.

▲throw0101c 6 hours ago | parent | next [-]

I.e.:

> An Extended Validation (EV) Certificate is a certificate conforming to X.509 that proves the legal entity of the owner and is signed by a certificate authority key that can issue EV certificates. EV certificates can be used in the same manner as any other X.509 certificates, including securing web communications with HTTPS and signing software and documents. Unlike domain-validated certificates and organization-validation certificates, EV certificates can be issued only by a subset of certificate authorities (CAs) and require verification of the requesting entity's legal identity before certificate issuance.

* https://en.wikipedia.org/wiki/Extended_Validation_Certificat...

▲organsnyder 5 hours ago | parent [-]

This was before Extended Validation was available. I don't remember the exact year, but I know for sure it was before 2007 (it was my dad's business, and he passed away in 2007).

▲tialaramex 5 hours ago | parent | prev | next [-]

Arguably in the extremely early days of SSL this is how it works, because the Certificate Authorities have never done this before and so sure, we fly our account excecutive out to your New York HQ and they sign (with a pen) a contract for $5000. Those early certificates are, by modern standards, worthless garbage, Chat GPT can probably teach you how to forge one though it might be too expensive for an amateur to do at home - but they do have street addresses and so on if you cared to read them. A web browser of course did not read those street addresses, and it has even less idea than you do whether the details correspond to reality.

Within a few years at most the Race to the Bottom is well underway - if I spent $10 on my "streamlined" process to offer 1000 certificates for $10 each I make more money than a rival spending $1000 to offer 100 certificates for $100 each. As a result "Domain Verification" is normal by the turn of the century. If you weren't here for the Dotcom boom, you have never seen any of this. If you bought books on this stuff on Amazon, or worse you've never read any books about it because it was all online, you arrived after DV was everywhere.

A peer mentions EV, that and OV are actually newcomers, the CAs realised that customers don't want to pay $100 for a "better" certificate when customers can't tell the difference so there began a meeting, the CA/Browser Forum sometimes just referred to as "cab" or "cabf". This meeting had two initial outputs. One of them endures to this day and only got stronger:

1. Certificate Authorities must meet at least certain Baseline Requirements (the "CA/BF BRs") for all certificates. You can't issue certificates which do not meet these requirements, putting a floor on how terrible your verifications can be.

2. Browser Vendors will add shiny UI for a new "Enhanced" verification certificate so that customers can see it's better and that's a new sales opportunity for CAs.

In practice after some years (2) stopped making sense. A shiny UI works on a desktop PC but there's no room on an early smartphone, so you just don't add this. Soon enough you make Desktop browsers look the same as your smartphone product and all the time your UX experts are telling you users do not undertand this UI (no surprise it was just a sales gimmick) and Safety experts are telling you that it's a net negative for user safety. So, bye bye EV UI in browsers.

But the other side of the bargain went very well. The BRs were a ratchet. In 2012 a ten year certificate issued based on some sales guy who met the director of a company at lunch was fine. By 2016 even five years is clearly too long and we should check some actual official records or have some kind of formal process. In 2012 it's fine to issue for mysql.test.internal because that's not a public name right? By 2016 uh, no, that's why it's not OK to issue that certificate, nobody could possibly own that name, do not issue bogus certificates, this is Serious Business. The BRs are how, gradually, we got where we are today.

▲mopsi 6 hours ago | parent | prev [-]

I went through the standard validation many times and the entire process felt like a security theater. For example, I had to upload a scan of my ID, but I know for a fact that in my jurisdiction no private entity can access government databases to validate that it is a genuine ID and not a photoshopped image. The government provides a SSO that third parties can use, but none of the main cert issuers implemented it, even for EV certs. They just trusted a JPEG.

It's like those arcane verification checks used by US-based services that ask for utility bills, as if they had any way of verifying an invoice issued by a small utility provider in rural Finland.

▲lb_ 4 hours ago | parent | prev | next [-]

The route hijack was RPKI valid because they spoofed Hetzner's AS into the AS_PATH and Hetzner used a max length of /24 on the ROA (https://bgp.tools/rpki-history?cidr=162.55.0.0%2f16&asn=2494...).

I personally believe the real fix would be to incentivise the use of CAA records, ideally including the accounturi parameter.

▲cowlevel 8 hours ago | parent | prev | next [-]

There's also the jabber.ru attack, the attacker physically intercepted the server's network cable and just issued themselves a cert.

▲woodruffw 9 hours ago | parent | prev | next [-]

70% adoption doesn't seem high enough or it to be a mandatory component of validation[1], that seems like it would be very disruptive to nearly a third of the Internet.

[1]: https://ipinfo.io/blog/rpki-protects-70-internet-routes

▲traceroute66 4 hours ago | parent | prev | next [-]

> RPKI validation as a "MUST" rather than piling on multiple sources of validation

And ASPA validation[1]

[1] https://blog.apnic.net/2026/07/09/aspa-at-apnic-strengthenin...

▲Lex-2008 9 hours ago | parent | prev | next [-]

I think having multiple authoritative DNS servers in different networks might help with this.

I'm not sure how CAs deal with situations when a domain has several authoritative DNS servers, but not all of them respond with expected reply. Would it would be a good idea for CAs to ask *all* authoritative DNS servers and require a coherent answer from them?

▲throw0101c 8 hours ago | parent [-]

> I think having multiple authoritative DNS servers in different networks might help with this.

For the case of the dns-01 challenge type, sure. But if you're using http-01 (or the attacker asks for it), then all those DNS servers will answer back with the same IP address for the A(AAA) query and you're back to the hijacking scenario.

▲Lex-2008 7 hours ago | parent | next [-]

> http-01

Good point, I forgot about that! But then CAA record might help, I guess, if you specify your accounturi?

▲throw0101c 6 hours ago | parent [-]

The attacker could hijack the network and initiate the ACME protocol against the CA in the CAA field. There is a draft for dns-account-01 which allows one to specify an ACME account which is authorized for updates (which uses the account key (and not cert key)):

* https://datatracker.ietf.org/doc/html/draft-ietf-acme-dns-ac...

▲Lex-2008 4 hours ago | parent [-]

haven't heard about dns-account-01, thanks! Heard only about dns-persist-01.

I meant to say that CAA record can also specify account, or at least Let’s Encrypt docs mention it[1]. So unless attacker has access to victim's account - they can't get a cert (unless they can hijack dns, too).

[1]: https://letsencrypt.org/docs/caa/#the-accounturi-parameter

▲throw0101c 3 hours ago | parent [-]

Account seems to be a later extension to CAA:

* https://datatracker.ietf.org/doc/html/rfc8657

▲codedokode 6 hours ago | parent | prev [-]

http-01 seems absolutely unreliable, any hosting provider or any government can issue fake certificates using it. What a garbage technology.

▲mixdup 4 hours ago | parent | next [-]

The point is to prove you control a domain. If you are answering traffic to requests from clients at that domain, you control it. Whether you're supposed to have control over it is outside the scope of this validation

▲Lex-2008 4 hours ago | parent | prev [-]

Agreed. On the other hand, these DV certificates only claim to verify who _controls_ the domain. Probably, you can agree that hosting provider and government can also control the domain, no? :)

Moreover, with dns-01, DNS provider can issue fake certificates - unless you self-host DNS - in which case, only your DNS registrar can... and a government.

▲throw0101c 3 hours ago | parent [-]

> Moreover, with dns-01, DNS provider can issue fake certificates - unless you self-host DNS - in which case, only your DNS registrar can... and a government.

Which is what DNSSEC is supposed to help with: the registrar/government would have to replace your DS/KSK with their own. Without DNSSEC queries/replies could potentially be altered over the wire (or you're in a race condition as to whether the real or attacker's reply get back first).

▲Ajedi32 7 hours ago | parent | prev | next [-]

Yeah, checking from multiple perspectives is a good idea, but still ultimately a band-aid over the top of a flawed system. A properly engineered solution would need to ultimately come down to cryptographic proof of domain ownership (whether through DNSSEC, or something similar). Even RPKI still seems a little too indirect for my tastes (it fixes BGP attacks but potentially leaves the door open to other forms of domain hijacking).

Maybe in practice this is good enough, but the security nerd inside me rebels at how sloppy the whole thing feels.

▲tptacek 6 hours ago | parent [-]

A lot of practical security is, in effect, band-aids over flawed systems. But I think it's useful to remember that the WebPKI doesn't actually want arbitrary secure DNS; it wants proof of domain ownership. There are ways to deliver that without forklifting in an entire new non-transparent PKI, especially since literally the only thing it will be used for is proving domain ownership.

▲Ajedi32 3 hours ago | parent [-]

I don't see how you can have actual "proof of domain ownership" without secure DNS, or some kind of similar PKI. Right now they don't check for proof of domain ownership at all, just whether you have temporary, de facto control of the DNS record or the network it points at. This usually works well enough, but won't do anything if a MITM attack occurs on the server side rather than the client side.

▲tptacek 2 hours ago | parent [-]

There's a whole design space of options here, one of the simpler ones being just checking via the registrar, using a protocol like RDAP. You could also use the DNS for this, but not require the entire zone to be encrypted (you'd be designing a new protocol, but that's what we have to do anyways, since DNSSEC is a dead letter).

▲Ajedi32 2 hours ago | parent [-]

Yes. That would fall under the "some kind of similar PKI" category I mentioned, as you'd still need some kind of PKI to secure the RDAP connection. (Perhaps that PKI could be simpler than full DNSSEC, but it would have to solve a lot of the same problems.)

▲tptacek 2 hours ago | parent [-]

For RDAP to work (for instance) you only need to bootstrap with secure connectivity to the registrars. Current DV challenges have a much harder problem: they have to work with every zone on the Internet. There's no real reason to accept that constraint!

When we think about "sound engineering" this is the level we need to start at, not "what features could we bolt on to core Internet infrastructure to define the problem away".

The people most avidly pushing DNSSEC for this problem aren't stupid. They're just smuggling in an additional premise: that the WebPKI itself is bad, and should be replaced with a DNS PKI. Seen through that lens, secure proof of domain ownership would be deeply problematic, as it would leave the DNS PKI with literally no reason to exist.

▲cowlevel 8 hours ago | parent | prev | next [-]

RPKI is, unfortunately, dual purpose as a government censorship mechanism. It would be trivial for the government of the Netherlands (or the USA, which the Netherlands is subservient to) to compel RIPE NCC to stop publishing RPKI records for Russian networks, and if RPKI was mandatory when this happened, this would cut Russia off the internet.

▲tomveber 8 hours ago | parent | prev [-]

[flagged]