Remix.run Logo
michaelt 6 days ago

> I've seen valid issued Google Trust certs floating around that were not authorized by the owner of the SAN being visited

Did you confirm with the owner that they were unauthorized?

And can you point to the certificates in the Certificate Transparency logs?

trod1234 6 days ago | parent [-]

> Did you confirm with the owner that they were unauthorized.

I confirmed with their support. I provided the certificate chain and sha-256 fingerprint being served, and they said it didn't match, and that they use a different provider for their certificates; which I suppose is Godaddy, at least that's what shows up on the crt.sh logs.

I don't run nor have access to a CT log for auditing. I was told it was revoked though. If you want to look into it you can; I'm including the CRT chain below.

There have been a number of issues uncovered while investigating the silent failing calls. Ranging from silent fail denial of service, unauthorized password changes after-the-fact, and with login credentials it seems some form of MITM translation, and these are consistent across many devices when accessing the site, or services.

The issues seem to clear up every month or so for about 1-2 weeks starting on the 4th, a new set of certs shows up every couple months.

The translation thing is that voip.ms doesn't allow @ symbols in passwords. About 2-4 hours after a lost password recovery the password that is set stops working with no change logged server-side. Replacing the token I used instead of @ with @, logs in without error from the edge successfully after that period occurs, despite their password policy/validator silent failing, and being against the use of that token which they have confirmed is still in effect. Craziness.

I can only conclude that this is some form MITM. I've seen similar issues across other vendors as well, but they haven't noticed failures yet, or have been completely non-responsive (with no phone contact), so they haven't been looking into it too hard, if at all.

www.voip.ms

-----BEGIN CERTIFICATE-----MIIDmjCCA0GgAwIBAgIRALnZP1MTVuRgEWRq2GuA7BkwCgYIKoZIzj0EAwIwOzELMAkGA1UEBhMCVVMxHjAcBgNVBAoTFUdvb2dsZSBUcnVzdCBTZXJ2aWNlczEMMAoGA1UEAxMDV0UxMB4XDTI1MDYwNjA2MzQxOFoXDTI1MDkwNDA3MzM1M1owEjEQMA4GA1UEAxMHdm9pcC5tczBZMBMGByqGSM49AgEGCCqGSM49AwEHA0IABHNo2vDB8rWItKKAgiIWPUU0T7upGdVUZE5uF24AjT9KmZhZBpdrXeOWJqWuA4jPWXBUzGrVzUGYsO6B/CvLkKqjggJNMIICSTAOBgNVHQ8BAf8EBAMCB4AwEwYDVR0lBAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAdBgNVHQ4EFgQUkFJpHoJsK+n+gV1HFtpEg27jxGAwHwYDVR0jBBgwFoAUkHeSNWfE/6jMqeZ72YB5e8yT+TgwXgYIKwYBBQUHAQEEUjBQMCcGCCsGAQUFBzABhhtodHRwOi8vby5wa2kuZ29vZy9zL3dlMS91ZGswJQYIKwYBBQUHMAKGGWh0dHA6Ly9pLnBraS5nb29nL3dlMS5jcnQwHQYDVR0RBBYwFIIHdm9pcC5tc4IJKi52b2lwLm1zMBMGA1UdIAQMMAowCAYGZ4EMAQIBMDYGA1UdHwQvMC0wK6ApoCeGJWh0dHA6Ly9jLnBraS5nb29nL3dlMS9fLTRpRndmQ2FjTS5jcmwwggEGBgorBgEEAdZ5AgQCBIH3BIH0APIAdwDd3Mo0ldfhFgXnlTL6x5/4PRxQ39sAOhQSdgosrLvIKgAAAZdEKXt9AAAEAwBIMEYCIQDjYC10JgSqWCbCE23l++70zgoHwTPUYsAf56DrZiWJdQIhANPwfZiTkV0N5eAVGYlRpPpQ88KovS80pPmThB8VHHzFAHcAfVkeEuF4KnscYWd8Xv340IdcFKBOlZ65Ay/ZDowuebgAAAGXRCl7agAABAMASDBGAiEAzfEhazBYmOhzSujGbLErjeTwKQvV3/ASvWENwXycXCoCIQDM+tYWt/xzqBcYd4Ivs2Pba/EIuBMhRY9Rq2CdntkqYDAKBggqhkjOPQQDAgNHADBEAiBzcp1G0vLRX+ZvWJFnRG83/pt+0fx4j1uXu66R4nbVyAIgekwYAEhhA7aJ19uykBfTG/wesrmcrkLxX6XjqEzE2L8=-----END CERTIFICATE----------BEGIN CERTIFICATE-----MIICnzCCAiWgAwIBAgIQf/MZd5csIkp2FV0TttaF4zAKBggqhkjOPQQDAzBHMQswCQYDVQQGEwJVUzEiMCAGA1UEChMZR29vZ2xlIFRydXN0IFNlcnZpY2VzIExMQzEUMBIGA1UEAxMLR1RTIFJvb3QgUjQwHhcNMjMxMjEzMDkwMDAwWhcNMjkwMjIwMTQwMDAwWjA7MQswCQYDVQQGEwJVUzEeMBwGA1UEChMVR29vZ2xlIFRydXN0IFNlcnZpY2VzMQwwCgYDVQQDEwNXRTEwWTATBgcqhkjOPQIBBggqhkjOPQMBBwNCAARvzTr+Z1dHTCEDhUDCR127WEcPQMFcF4XGGTfn1XzthkubgdnXGhOlCgP4mMTG6J7/EFmPLCaY9eYmJbsPAvpWo4H+MIH7MA4GA1UdDwEB/wQEAwIBhjAdBgNVHSUEFjAUBggrBgEFBQcDAQYIKwYBBQUHAwIwEgYDVR0TAQH/BAgwBgEB/wIBADAdBgNVHQ4EFgQUkHeSNWfE/6jMqeZ72YB5e8yT+TgwHwYDVR0jBBgwFoAUgEzW63T/STaj1dj8tT7FavCUHYwwNAYIKwYBBQUHAQEEKDAmMCQGCCsGAQUFBzAChhhodHRwOi8vaS5wa2kuZ29vZy9yNC5jcnQwKwYDVR0fBCQwIjAgoB6gHIYaaHR0cDovL2MucGtpLmdvb2cvci9yNC5jcmwwEwYDVR0gBAwwCjAIBgZngQwBAgEwCgYIKoZIzj0EAwMDaAAwZQIxAOcCq1HW90OVznX+0RGU1cxAQXomvtgM8zItPZCuFQ8jSBJSjz5keROv9aYsAm5VsQIwJonMaAFi54mrfhfoFNZEfuNMSQ6/bIBiNLiyoX46FohQvKeIoJ99cx7sUkFN7uJW-----END CERTIFICATE----------BEGIN CERTIFICATE-----MIICCTCCAY6gAwIBAgINAgPlwGjvYxqccpBQUjAKBggqhkjOPQQDAzBHMQswCQYDVQQGEwJVUzEiMCAGA1UEChMZR29vZ2xlIFRydXN0IFNlcnZpY2VzIExMQzEUMBIGA1UEAxMLR1RTIFJvb3QgUjQwHhcNMTYwNjIyMDAwMDAwWhcNMzYwNjIyMDAwMDAwWjBHMQswCQYDVQQGEwJVUzEiMCAGA1UEChMZR29vZ2xlIFRydXN0IFNlcnZpY2VzIExMQzEUMBIGA1UEAxMLR1RTIFJvb3QgUjQwdjAQBgcqhkjOPQIBBgUrgQQAIgNiAATzdHOnaItgrkO4NcWBMHtLSZ37wWHO5t5GvWvVYRg1rkDdc/eJkTBa6zzuhXyiQHY7qca4R9gq55KRanPpsXI5nymfopjTX15YhmUPoYRlBtHci8nHc8iMai/lxKvRHYqjQjBAMA4GA1UdDwEB/wQEAwIBhjAPBgNVHRMBAf8EBTADAQH/MB0GA1UdDgQWBBSATNbrdP9JNqPV2Py1PsVq8JQdjDAKBggqhkjOPQQDAwNpADBmAjEA6ED/g94D9J+uHXqnLrmvT/aDHQ4thQEd0dlq7A/Cr8deVl5c1RxYIigL9zC2L7F8AjEA8GE8p/SgguMh1YQdc4acLa/KNJvxn7kjNuK8YAOdgLOaVsjh4rsUecrNIdSUtUlD-----END CERTIFICATE-----

SHA-256 Fingerprint:

FB:4E:10:D3:58:0A:01:1A:9E:82:92:5B:33:AE:1C:E3:6D:5C:B3:97:53:73:B4:1C:4A:7E:30:8B:49:44:BA:24

Support staff said they were investigating the issue, but its been almost 90 days now without next-steps, explanation, or anything actionable. I've been getting stonewalled for quite awhile now.

I've seen this enough times now recently that TLS doesn't seem trustworthy anymore. Its quite maddening too where at a fairly fundamental level in troubleshooting; what you see on one end isn't what is actually being hosted on the other.

SahAssar 5 days ago | parent [-]

The cert you mention is this one, right? https://crt.sh/?id=18844641499

Seems like they use cloudflare as their DNS provider, which uses Google as their cert provider and this has happened before with them. See for example https://news.ycombinator.com/item?id=40452307 where I got into the same discussion but where it was due to porkbun using cloudflare as their DNS backend.

I would not treat this as TLS being untrustworthy, I would treat it as cloudflare issuing certs for you even if you just want to use their DNS (and not their WAF or other products).

trod1234 5 days ago | parent [-]

If that is the one that matches what was posted then yes. A cursory glance, those fingerprints match so I'd say yes that is one of the certificates with which we've narrowed issues down to.

I would think that a large company like voip, would have their certificate provider documented, and available to check when there is a significant issue, so when their customers report a problem and they say it isn't a match that's exactly what they mean.

Also, the only indicator of any of these issues which prompted all this, with any real explanation, is with the cert and by extension the secure tunnel which cannot be trusted. The issues extend to not just this one vendor, but several others as well across multiple devices and network connections. The translation issue appears only visible with this provider though due I suspect to their non-standard password policy, which appears contradictory at the edge in function.

Saying TLS is trustworthy, where things that shouldn't ever happen under TLS guarantees are happening, with no viable alternative explanation for the issues, where they have been troubleshooted over months at both ends, including all the way down to the raw physical level of the OSI level for traffic (at least at the edge)... that doesn't leave anyone with anywhere to go.

Still Trust TLS? If there were a reasonable alternative explanation that ties in and touches on all the issues both mentioned and unmentioned, I'd be the first to consider it.

Clearly there are objective issues where service cannot be relied upon for a business, let alone for anything less demanding. The issues are also not vendor specific and seem to be coupled loosely to geographical region. The only commonality are these Google Trust certificates.

Communications services fail silently across multiple providers, contact forms either fail to submit with weird HTTP error codes for large providers or submit with success only to have non-response with no verifiable record of submission after-the-fact, support chat's fail to load or load with a chatbot pretending to be a human with no record after-the-fact, emails disappear, and many other things that effectively rely upon only one thing in common when taken in aggregate.

When its one thing that happens in isolation at a single vendor sure I'd be more receptive to it being something else on the vendor side, but when every single path fails regularly in the same chaotic way in narrow time horizons, there's a significant issue, and one must question not only the guarantees, but the only common links.

Three or more path failures related to communication, within a short time horizon, all leading back to TLS guarantees, is beyond an astronomical bayes probability that something there is silently happening over those links that shouldn't be happening.

SahAssar 5 days ago | parent [-]

The TLS guarantees are to the edge of the infra of the vendor. If that vendor has decided to use infra providers that issue certs for them without their knowledge and they have not implemented CAA then the blame is not on TLS, it is on the vendor. A lot of what you mention can be explained by cloudflare issuing certs for customers without them knowing when using their DNS, an agressive WAF or other much more plausible things.