| |
| ▲ | hnfong 7 hours ago | parent | next [-] | | Yes. As far as I understood it, the result was an incoherent mess. I think some characters got different code points, while others were merged. And apparently the Japanese complained bitterly over the ones that were merged. If you had read any articles about that, this is probably what you have in mind right now. And I'm also a bit tilted by the ones that had different code points, because when processing CJK text now we have to deal with characters that are (in my native Cantonese) essentially the same, looks similar (to my undiscriminating eyes), yet having different code points so that things like text search sometimes don't work. Of course I'm not "blaming" the Japanese, if anything the simplified vs traditional Chinese thing is much more of a practical problem, and the conflicting code points I deal with on a routine basis are more of a Hong Kong vs Taiwan thing, but I was told that the Unified CJK thing adopted a different "philosophy" from the rest of Unicode (which I think really is some kind of Aristotelian essentialism...) mostly due to vocal objections from the Japanese. | | |
| ▲ | wodenokoto 5 hours ago | parent | next [-] | | Han-unification was the result of Asian scholars, in particular the Ideaographic Research Group [1] I have no idea what "Aristotelian essentialism" is supposed to mean, or if you are saying that the unification was that. > And apparently the Japanese complained bitterly over the ones that were merged. and > but I was told that the Unified CJK thing adopted a different "philosophy" from the rest of Unicode [...] mostly due to vocal objections from the Japanese. Seems to contradict each other. [1] https://en.wikipedia.org/wiki/Ideographic_Research_Group | | |
| ▲ | hnfong 5 hours ago | parent [-] | | Sorry, probably not very standard terminology. I somehow picked it up from the OReilly book: Fonts & Encodings: From Advanced Typography to Unicode and Everything in Between Page 55: """ there are characters with no glyphs. glyphs that can correspond to a number of different characters according to context. Glyphs that correspond to multiple characters at the same time (with weightings assigned to each), and even more possibilities. The problem of glyphs and characters is so complex that it has gone beyond the realm of computer specialists and has come to be of interest even to philosophers. For example, the Japanese philosopher Shigeki Moro, who has worked with ideographic characters in Buddhist documents, goes so far in his article Surface or Essence: Beyond Character Model Set [274] as to say that Unicode's approach is Aristotelian essentialist and to recommend supplanting it by an approach inspired by Jacques Derrida's theory of writing [114, 115]. The reader interested in the philosophical aspects of the issue is invited to consult [165,156], in addition to the works cited above. """ I think "essentialist" is probably a good description of the philosophy of how Unicode defines characters as opposed to fonts and glyphs, so I adopted it. |
| |
| ▲ | pamcake 6 hours ago | parent | prev | next [-] | | > I was told that the Unified CJK thing adopted a different "philosophy" from the rest of Unicode What would that philosophy be about? Sounds apocryphal. Unicode has never done "unification" like that for other languages/scripts? i/ı/i, ö/ø/ø̈/oͤ: Same same, different codepoints. Search and sorting is a mess everywhere. Depending on your locale, ö sorts either after o or after z. Sometimes it's semantically and phonetically equivalent to o wrt search but moreoften not. https://en.wikipedia.org/wiki/%C3%96 | | |
| ▲ | yorwba 5 hours ago | parent | next [-] | | Well, Unicode didn't unify 門/门 either, but the appearance of the second glyph changes a lot depending on whether you use a Chinese font or a Japanese one. Meanwhile they did unify all the graphical variants of Latin characters with identical semantics, it's just that everyone is so used to the result that you hardly notice not being able to control whether g has a hook or a looped tail, and nobody expects to copy some German blackletter text into a French document and have it show up visually distinct, without explicitly setting the font. | | |
| ▲ | Joker_vD 4 hours ago | parent [-] | | Yes, and Cyrillic letters also look quite different in Bulgarian and Russian fonts. So what? |
| |
| ▲ | wl 35 minutes ago | parent | prev | next [-] | | Unicode has done the same unification for cuneiform languages. If you’re dealing with texts in Hittite, Sumerian, or Akkadian, you need to make sure you’re using the right fonts, otherwise the text will look entirely wrong. | |
| ▲ | wodenokoto 5 hours ago | parent | prev | next [-] | | Same thing, different code points is the opposite of han unification. Han unification is same code point, different thing. | |
| ▲ | numpad0 4 hours ago | parent | prev [-] | | Unicode merges some Chinese and Japanese characters into same code points and leave others as they are, on random per-character basis. So Chinese guys gets a ton of Japanese mixed texts and it causes tons of problems. Japanese users are angry when system chooses Chinese fonts over Japanese, Chinese users apparently suffer with a font mishmash of Chinese and Japanese all the time like a mid-word capitalization. The official sanctioned solution is to just commit a genocide and nuke the offending language out of the system or to attach IVS to every Japanese characters which is like zero padding every single characters by a byte or two. It's really putting everyone in pain and making absolutely no one happy, which sounds like what a good compromise tends to be, but LLMs seem to be struggling with there being multiple completely separate language families sharing codepoints but not character shapes or syntax. I've seen a smaller LLM model use made up verbs spilling over from the other language. I believe ML guys don't like datasets that do that kind of things. Same apply to image models. |
| |
| ▲ | numpad0 4 hours ago | parent | prev | next [-] | | I don't know but last I glanced at it, the Unification was implied to be coming from Chinese pressure than Japanese. Each of Kanji/Hanzi countries already had each own acceptable common use character sets and those could have been just populated into each own spaces. That seemed like the original plan at some point. There has to be a reason why it happened(as to why Unicode suddenly started insisting it has to fit inside a 2^16 total chars or whatever). | |
| ▲ | panzi 5 hours ago | parent | prev [-] | | Interesting. I guess there are no well working normalization functions as there are for the different code points that can make umlauts (äöü)? |
| |
| ▲ | muvlon 5 hours ago | parent | prev [-] | | Meanwhile, Latin a and Cyrillic а, which look identical, have the same origin and are often pronounced identically too, get separate codepoints. CJK unification was not based on any clear preestablished principles but simply an attempt to cling to 16-bit codepoints, Asian languages be damned. | | |
| ▲ | esrauch 4 hours ago | parent [-] | | Cling to 16 bit codepoints sounds like a legitimate explanation? Latin and Cyrillic didn't would have saved no relevant amount of space. It looks like the CJK unified space is over 20,000 characters, so that's a real technical magnitude distinction compared to Latin and Cyrillic. "Asian languages be damned" seems like a bad faith read, compared to "Java and Windows char is 16 bits and that will never change realistically" (and in fact they still haven't, even in 2026 things which rely on UTF16 instead of UCS2 are still commonly bugged unfortunately) | | |
| ▲ | Joker_vD 3 hours ago | parent [-] | | We also have at least 15 copies of Latin alphabet in Unicode, all differently styled/fonted (there is chancery cursive, mathematical cursive, the fraktur version, the full-width version, etc.) | | |
| ▲ | gucci-on-fleek 3 hours ago | parent [-] | | Yes, but most of those are intended to be used exclusively for math, where the different styles do have semantic meaning. Even in introductory textbooks, it's pretty common to see the roman, calligraphic, and double-struck variants of a single letter all used with different meanings. Plus, these characters were only added 20+ years later, and in the supplemental planes, not the BMP. | | |
| ▲ | Joker_vD 3 hours ago | parent [-] | | Letterlike Symbols and Fullwidth Forms are in BMP (and I generously consider Enclosed Alphanumerics to not be a copy of a Latin alphabet). | | |
| ▲ | gucci-on-fleek an hour ago | parent [-] | | Letterlike Symbols is a bit of a weird grab bag of characters, most of which are clearly distinct from their "regular" Latin versions, but I do agree that a few of them are pointless exact copies of regular Latin letters. The fullwidth/halfwidth stuff is a bit of a mess, but you could argue that these are actually CJK characters that just happen to resemble Latin characters (much like how Greek and Cyrillic both happen to have letters that look a lot like the Latin "A"), since they only exist for compatibility with older CJK encodings. This wouldn't be a very good argument though :) |
|
|
|
|
|
|