In practice it's much easier than you seem to have understood

The names PKIX writes into dnsName are exactly the same as the hostnames in DNS. They are defined to always be Fully Qualified, and yet not to have a trailing dot, you don't have to like that but it's specified and it's exactly how the web browsers worked already 25+ years ago.

You're correct that they're not the on-wire label-by-label DNS structure, but they are the canonical human readable DNS name, specifically the Punycode encoded name, so [the website] https://xn--j1ay.xn--p1ai/ the Russian registry which most browsers will display with Cyrllic, has its names stored in certificates the same way as it is handled in DNS, as Punycode "xn--j1ay.xn--p1ai". In software I've seen the label-by-label encoding stuff tends to live deep inside DNS-specific code, but the DNS name needed for comparing with a certificate does not do this.

You don't need to "deal with" case except in the sense that you ignore it, DNS doesn't handle case, the dnsName in SANs explicitly doesn't carry this, so just ignore the case bits. Your DNS client will do the case bit wiggling entropy hack, but that's not in code the certificate checking will care about.

You do need to care about wildcards, but we eliminated the last very weird certificate wildcards because they were minted only by a single CA (which argued by their reading they were obeying PKIX) and that CA is no longer in business 'cos it turns out some of the stupid things they were doing even a creative lawyerly reading of specifications couldn't justify. So the only use actually enabled today is replacing one DNS label at the front of the name. Nothing else is used, no suffixes, no mid-label stuff, no multi-label wildcards, no labels other than the first.

Edited to better explain the IDN situation hopefully

▲

cryptonector 2 days ago | parent [-]

> You don't need to "deal with" case except in the sense that you ignore it, DNS doesn't handle case

The DNS is case-insensitive, though only for ASCII. So you have to compare names case-insensitively (again, for ASCII). It _is_ possible to have DNS servers return non-lowercase names! E.g., way back when sun.com's DNS servers would return Sun.COM if I remember correctly. So you do have to be careful about this, though if you do a case-sensitive, memcmp()-like comparison, 999 times out of 1,000 everything will work, and you won't fail open when it doesn't.

	▲	tialaramex 2 days ago \| parent [-]
		You're correct that your DNS queries and answers can set the case bit, but the protocol design always said that while the answers must match the query, the case isn't actually significant. For a long time that's just an obscure Um Actually nerd trivia question, but traditional (before DPRIVE) DNS is very old and is mostly a UDP protocol, so people try to spoof it, let's follow that story: At first: The only anti-spoof defence provided in DNS is an ID number, initially people are like 1, 2, 3, 4, 5... and so the attacker watches you use these IDs then makes you ask a question, "6: A? mybank.example" and simultaneously they answer "6: A 10.20.30.40" before your real DNS server could likely respond - choosing their own IP address. Six is the expected ID, you just got spoofed and will visit their fake bank. So then: DNS clients get smarter using randomisation for the ID, 23493, 45390, 18301... this helps considerably, but bandwidth is cheap and those IDs are only 16-bit so a bad guy can actually send you 65536 answers and get a 100% success rate with spoofing, or more realistically maybe they send 1000 answers and still have more than 1% success. Today as a further improvement, we use Paul Vixie's bit 0x20 hack which uses every letter of a DNS label to hide one bit of entropy in the case in addition to the random ID. Now the attacker has to try not only different IDs but different case spellings, A? mYbANk.eXAmpLE or maybe A? MyBANk.EXAMple or A? mybaNK.EXamPLE -- only responses with the right case match, the others are ignored. So, because of all this security shenanigans, your DNS client knows that case in DNS queries doesn't matter and will do what we want for this purpose. [Edited: fixed a few typos]