Remix.run Logo
amluto 5 hours ago

This is kind of amazing. I'm suspicious that the site operator has absolutely no idea what they're doing.

> DoD Cyber Exchange site is undergoing a TSSL Certification renewal

I'm imagining someone searching around for a consulting or testing company that will help them get a personal TSSL Certification, whatever that is (a quick search suggests that it does not exist, as one would expect). And perhaps they have no idea what TLS is or how any modern WebPKI works, which is extra amazing, since cyber.mil is apparently a government PKI provider (see the top bar).

Of course, the DoD realized that their whole web certificate system was incompatible with ordinary browsers and they wrote a memo (which you have to click past the certificate error to read):

https://dl.dod.cyber.mil/wp-content/uploads/pki-pke/pdf/uncl...

saying that, through February 2024, unclassified DoD sites are permitted to use ordinary commercial CAs.

If the DoD were remotely competent at this sort of thing, they would (a) have CAA records (because their written policy does nothing whatsoever to tell the CA/B-compliant CAs of the world not to issue .mil certificates, (b) run their own intermediate CA that had a signature from a root CA (or was even a root CA itself), and (c) use automatically-renewed short-lived certificates for the actual websites.

cyber.mil currently uses IdenTrust, which claims to be DoD approved. They also, ahem, claim to support ACME:

> In support of the broader CA community, IdenTrust—through HID and the acquisition of ZeroSSL—actively contributes to the development and maintenance of major open-source ACME clients, including Caddy Server and ACME.sh. These efforts help promote accessibility, interoperability, and automation in certificate management.

Err... does that mean that they actually support ACME on their DoD-approved certificates or does that mean that they bought some companies that participate in the ACME ecosystem? (ACME is not amazing except in contrast to what came before and as an exercise in getting something reasonable deployed in a very stodgy ecosystem, but ACME plus a well-designed DNS-01 implementation plus CAA can be very secure.)

The offending certificate is:

    Certificate:
        Data:
            Version: 3 (0x2)
            Serial Number:
                40:01:95:b4:87:b3:a3:a9:12:e0:d7:21:f8:b3:91:61
            Signature Algorithm: sha256WithRSAEncryption
            Issuer: C=US, O=IdenTrust, OU=TrustID Server, CN=TrustID Server CA O1
            Validity
                Not Before: Mar 20 17:09:07 2025 GMT
                Not After : Mar 20 17:08:07 2026 GMT
            Subject: C=US, ST=Maryland, L=Fort Meade, O=DEFENSE INFORMATION SYSTEMS AGENCY, CN=public.cyber.mil
At least the site uses TLS 1.3.
BigTTYGothGF 4 hours ago | parent | next [-]

> If the DoD were remotely competent at this sort of thing

That's probably one of the things they were forced to contract out.

stult 2 hours ago | parent | prev | next [-]

There are a few reasons DoD PKI is a shitshow which make it somewhat more understandable (although only somewhat).

First, the issues you describe affect only unclassified public-facing web services, not internal DoD internet services used for actual military operations. DoD has its own CA, the public keys for which are not installed on any OS by default, but anyone can find and install the certs from DISA easily enough. Meaning, the affected sites and services are almost entirely ones not used by members of the military for operational purposes. That approach works for internal DoD sites and services where you can expect people to jump through a couple extra hoops for security, but is not acceptable for the general public who aren't going to figure out how to install custom certs on their machine to deal with untrusted cert errors in their browser. That means most DoD web infra is built around their custom PKI, which makes it inappropriate for hosting public sites. Thus anyone operating a public DoD site is in a weird position where they deviate from DoD general standards but also aren't able to follow commercial standard best practices without getting approval for an exception like the one you linked to. Bureaucratically, that can be a real nightmare to navigate, even for experienced DoD website operators, because you are way off the happy path for DoD web security standards.

Second, many DoD sites need to support mTLS for CAC (DoD-issued smartcards) authentication. That requires the site to use the aforementioned non-standard DoD CA certs to validate the client cert from the CAC, which in turn requires that the server's TLS cert be issued by a CA in the same trust chain, which means the entire site will not work for anyone who hasn't jumped through the hoops to install the DoD CA certs. Meaning, any public-facing site has to be entirely segregated from the standard DoD PKI system. For now, that means using commercial certs, which in turn requires a vendor that meets DoD supply chain security requirements.

Third, most of these sites and services run on highly customized, isolated DoD networks that are physically isolated from the internet. There's NIPR (unclassified FOUO), SIPR (classified secret), and JWICS (classified top secret). NIPR can connect to the regular internet, but does so through a limited number of isolated nodes, and SIPR/JWICS are entirely isolated from the public internet. DoD cloud services are often not able to use standard commercial products as a result of the compatibility problems this isolation causes. That puts a heavy burden on the engineers working these problems, because they can't just use whatever standard commercial solutions exist.

Fourth, the DoD has only shifted away from traditional old school on-prem Windows Server hosting for website to cloud-hosting over the past few years. That has required tons of upskilling and retraining for DoD SREs, which has not been happening consistently across the entire enterprise. It also has made it much harder to keep up with the standards in the private sector as support for on-prem has faded, while the assumptions about cloud environments built into many private sector solutions don't hold true for DoD.

Fifth, even with the move to cloud services, the working conditions can be so extraordinarily burdensome and the DoD-specific restrictions so unusual, obscure, poorly documented, and difficult to debug that it dramatically slows down all software development. e.g., engineers may have to log into a jump box via a VDI to then use Jenkins to run a Groovy script to use Terraform to deploy containers to a highly customized version of AWS.

Ultimately, the sites this affects are ones which are lower priority for DoD because they are not operationally relevant, and setting up PKI that can easily service both their internal mTLS requirements and compatibility with commercial standards for public-facing sites and services is not totally straightforward. That said, it is an inexcusable shitshow. Having run CAC-authenticated websites, I can tell you it's insane how much dev time is wasting trying to deal with obscure CAC-related problems, which are extremely difficult to deal with for a variety of technical and bureaucratic reasons.

master_crab an hour ago | parent | next [-]

Good write up. Not to mention the other big HR constraints on DoD engineers: they almost always have to be a “US person.”

Anyone who gets a CAC working on a personal computer deals with this all too much. The root certs DoD uses are not part of the public trusted sources that commonly come installed in browsers.

stult an hour ago | parent [-]

lol I very nearly included a rant about that but decided it was too far off topic. Not being able to smoke weed may be more of an obstacle these days though.

amluto an hour ago | parent | prev | next [-]

> That requires the site to use the aforementioned non-standard DoD CA certs to validate the client cert from the CAC, which in turn requires that the server's TLS cert be issued by a CA in the same trust chain, which means the entire site will not work for anyone who hasn't jumped through the hoops to install the DoD CA certs. Meaning, any public-facing site has to be entirely segregated from the standard DoD PKI system. For now, that means using commercial certs, which in turn requires a vendor that meets DoD supply chain security requirements.

Is this actually all the way technically correct? As far as I know, there is no requirement that the trust chains for server certificates and client certificates are in any way related. It seems to me that it would be perfectly possible for the DoD to use its own entirely private client certificate infrastructure but to still have the server certificate use something resembling an ordinary root certificate.

This is not to say that this would actually be all that worthwhile.

mpyne 29 minutes ago | parent [-]

> Is this actually all the way technically correct? As far as I know, there is no requirement that the trust chains for server certificates and client certificates are in any way related. It seems to me that it would be perfectly possible for the DoD to use its own entirely private client certificate infrastructure but to still have the server certificate use something resembling an ordinary root certificate.

I think you're right that it's possible in principle for a Web server to enforce use of DoD CAC (enforcing the client cert being in the DoD PKI) without itself using a DoD PKI cert on the server side.

That said there's little benefit to it, users who haven't jumped through hoops to install DoD root CA certs won't typically be able to get their browsers to present them to the remote server in the first place, and if we're willing to jump through those hoops then there's no good reason for the DoD server not to have a DoD PKI cert.

amluto 18 minutes ago | parent [-]

I've never used one of the DoD smartcards, but I can certainly imagine the DoD wanting a user of one of these smartcards to be able to use it with a COTS client device to authenticate themselves.

robolange 2 hours ago | parent | prev | next [-]

> engineers may have to log into a jump box via a VDI to then use Jenkins to run a Groovy script to use Terraform to deploy containers to a highly customized version of AWS.

This hits too close to home. I'm sending you my therapist's bill for this month.

wbl 29 minutes ago | parent | prev | next [-]

A site can authenticate a client cert from a different PKI than one it has.

notesinthefield 2 hours ago | parent | prev [-]

I do lots of fed cloud work and im stealing this comment to use during the next conversation around why on-prem DoD PKI is now a huge PIA.

4 hours ago | parent | prev | next [-]
[deleted]
5 hours ago | parent | prev | next [-]
[deleted]
charcircuit 4 hours ago | parent | prev [-]

I think you underestimate the number of people who accidentally have their https carts expire. Instead of blaming the people running these systems on why they let it expires, it would be more productive to improve the system to make this less likely to happen.

alexjplant 3 hours ago | parent | next [-]

ACME [1] has been a thing for more than 10 years and has been a stable specification for 7 years. There were similar vendor-specific implementations that preceded it. The DoD has employed none of these solutions for their flagship infosec public web presence. If they were going to automate this then they surely would have done so by now. The reasons why are opaque but people who have experience working in this space might be able to make an educated guess.

[1] https://en.wikipedia.org/wiki/Automatic_Certificate_Manageme...

charcircuit 3 minutes ago | parent [-]

It may be a thing, but it is not mandatory and issues can still happen that cause the automatic renewal to fail. There still exist holes where someone can have a cert for their site expire.

JoshTriplett 3 hours ago | parent | prev | next [-]

Which is exactly what has happened, with an automated protocol for certificate renewal.

charcircuit 3 minutes ago | parent [-]

Even with that existing there are still holes with that solution as it's still happening.

RIMR 42 minutes ago | parent | prev [-]

Look, when I forget to renew the cert on my Jellyfin server, like 4 people suffer.

When the DoD forgets to renew the cert for their cybersecurity download website AND can't figure what a A TLS cert even is (calling it a "TSSL Certification"), this is an indicator that our military has absolutely zero understanding of the most basic cybersecurity concepts.

If you can't tell the difference between a hobbyist forgetting to renew their Let's Encrypt cert, vs. a trillion-dollar military not even knowing what a certificate is, maybe you should work for our military, because they can't tell the difference either.