Remix.run Logo
pwdisswordfishz a day ago

Or unpaired surrogates. Or unassigned code points. Or fullwidth characters. Or "mathematical bold" characters. Though the latter two should be probably solved with NFKC normalization instead.

chrismorgan 13 hours ago | parent [-]

> Or unpaired surrogates.

That’s just an invalid Unicode string, then. Unicode strings are sequences of Unicode scalar values, not code points.

> unassigned code points

Ah, the tyranny of Unicode version support. I was going to suggest that it could be reasonable to check all code points are assigned at data ingress time, but then you urgently need to make sure that your ingress system always supports the latest version of Unicode. As soon as some part of the system goes depending on old Unicode tables, some data processing may go wrong!

How about Private Use Area? You could surely reasonably forbid that!

> fullwidth characters

I’m not so comfortable with halfwidth/fullwidth distinctions, but couldn’t fullwidth characters be completely legitimate?

(Yes, I’m happy to call mathematical bold, fraktur, &c. illegitimate for such purposes.)

> solved with NFKC normalization

I’d be very leery of doing this on storage; compatibility normalisations are fine for equivalence testing, things like search and such, but they are lossy, and I’m not confident that the lossiness won’t affect legitimate names. I don’t have anything specific in mind, just a general apprehension.