UTS #39: Unicode Security Mechanisms

Posted by robin_reala 3 days ago

Counter48Comment31OpenOriginal

Comments

Comment by jjmarr 3 days ago

Banning the "limited use" category by default in the "General Security Profile" is interesting given it overrepresents indigenous scripts.

For example, Canadian Aboriginal Syllabics is the writing system of Inuktitut, the primary language of the Canadian territory of Nunavut spoken by 63.1% of the population. It seems like they're being disenfranchised by this proposal.

Comment by rurban 3 days ago

I compiled an overview who uses it, and who should: https://github.com/rurban/libu8ident/blob/master/doc/c11.md

Comment by vlfiny 2 days ago

[flagged]

Comment by i18nagentai 3 days ago

[flagged]

Comment by TacticalCoder 3 days ago

[flagged]

Comment by bombcar 3 days ago

The reality is that unless your in the US or "most" of Europe, you're going to WANT Unicode domain names - people in china are mostly fine with http://www.china.cn but they're going to really want to use their own language https://xn--fiq8ituh5mn9d1qbc28lu5dusc.xn--fiqs8s or https://xn--mgbeiue0iwb.xn--ngbc5azd - https://xn--wgv71a119e.jp

Demanding that everyone use our alphabet because we're too dumb to recognize the difference between google.com and xn--ggle-55da.com is the worst kind of "security through superior firepower"

(the fact that HN shows these correctly in the EDIT window but have to dump to punycode in display is sad, tbh)

Comment by duskwuff 3 days ago

Not even "most" of Europe - almost all European languages use diacritics to some extent or another. In the context of domain names, some words are comprehensible with the diacritics stripped, but there's some loss of precision. (For instance, an "ñ" is critical if you're trying to say "happy new year" in Spanish...)

Comment by octoberfranklin 3 days ago

UCS-2 (i.e. Javascript, 16-bit character space) has enough codepoints for every language still spoken except Chinese (even after undoing Han Unification, which was dumb).

Unicode is China forcing variable-length character encodings on the rest of the world.

Comment by rerdavies 3 days ago

If you really must blame anyone, Japan bears significantly more blame. (Tongue-out Smiley Face Emoji). Plus the original Unicode Consortium for being insufficiently sensitive to requirements for Asian scripts in the first place.

A summary of a fascinating rabbit hole. Here are the scripts that are included in the UTF Supplementary Multilingual Plane, which cannot be encoded in UCS-2, excluding scripts of primarily historical and archeological interest. Obviously, they can all be encoded in UTF-16.

Deseret - The Mormon Alphabet, designed by Joseph Smith to remove corruption in English when written in the Latin alphabet. Used briefly in the 19th century, but no longer actively used, except by a small community of hobbyists and scholars.

Shavian - A phonetic alphabet for English, whose development was funded by George Bernard Shaw. Currently only used by hobbyists.

Osmanya - One of several competing alphabets for the Somali language. Officially superseded by the Somali Latin alphabet in 1972. "Material written in the script is 'almost non-existent'" -- Wikipedia

Adlam - Use to write the Fuldani language. Spoken by about 36m people. The Adlam script is currently taught in Fuldani schools, and is used widely.

Hanifi Rohingya - One of three scripts currently actively used to write Rohingya, the others being Rohingya Arabic, and Rohingya Latin. Rohingya is spoken by 1.5m people. There are at least two newspapers published in Hanifi Rohingya. It is often used in the Rohingya diaspora, and widely taught in Rohingya refugee camps.

Osage - Currently, the official alphabet of the Osage language. Invented in 2006. Osage is spoken by about 50,000 people, and children are taught to use the Osage alphabet in schools.

Medefaidrin - a constructed script and language created as a Christian sacred language by an Ibibio congregation in 1930s Nigeria. It has its roots in glossolalia ('speaking in tongues').

Comment by colejohnson66 2 days ago

There's a lot of languages in plane 1: https://en.wikipedia.org/wiki/Plane_(Unicode)#Supplementary_...

Comment by monster_truck 3 days ago

this might be the dumbest thing I've read on HN in 2026

Comment by octoberfranklin 3 days ago

It's the facts, man.

If you want to say that the facts are dumb, well, you can do that.

Comment by gnabgib 3 days ago

No, it's not. UTF-16 supports the same characters as UTF-32 and UTF-8 (all three support all of Unicode). UTF-16 does that by having... variable width characters, in the form of 2-UTF16 code units for some chars.

UTF-16 wastes a bunch of space (when you don't need 16 bits to mark a character), and doesn't avoid variable-width encoding. UTF-8 tries to optimize space (but is ASCII skewed) with more variance (but once you can't infer length from bytes, what's the difference?) UTF-32 is a space hog at the benefit of predictable length encoding (a full Int32 to represent the null char, or a carriage return)

Comment by octoberfranklin 3 days ago

Congratulations, you've discovered why UTF-16 is stupid. You pay the space penalty of two-byte characters and the complexity penalty of variable-length characters.

It's the worst of both worlds!

Seriously nobody advocates UTF-16. The smart people advocate UCS-2.

Comment by gnabgib 3 days ago

You can either have Javascript uses UTF16 because UCS2 is dead. Or you can have variable width is stupid. JavaScript supports UTF16 (otherwise; no emoji). UCS2 is also a Unicode standard - just an old one, in their own words:

> In particular, for the purposes of data exchange, UCS-2 and UTF-16 are identical formats.

https://web.archive.org/web/20030818043641/http://www.unicod...

Comment by rerdavies 3 days ago

Ok. lol. THAT's the dumbest thing I've read on HN.

Comment by tialaramex 3 days ago

The insistence that Javascript means UCS-2 not UTF-16 involves selectively reading a version of the standards document that's more than a decade obsolete.

If you don't like facts that is dumb.

Comment by rurban 2 days ago

We ended up there because committees were afraid to say no to requests adding chinese, japanese or whatever variable names to your favorite compiler. But then they also didn't care about identifiers staying identifiable, which is the goal of TR 39. So many compilers did add "Unicode support" for the language.

Only 3-4 compilers stayed secure in this regard, by either disallowing unicode names, or by using TR39.

What else is a high security riscs? All names, such as path names or user names on OS kernels, like filesystems. Filesystem or user db's with unicode support but without identifier support such as TR39 can also lead to security attacks. Apple once had at least HPFS which did normalization, but the recently gave up, because nobody cared. And Linus still calls garbage in, garbage out valuable. Even if the famous ssh attack used such an homoglyph attack to stay undetected.

Comment by Georgelemental 3 days ago

Unicode is complex because human language is complex. If you want someone to blame, blame God:

> Go to, let us go down, and there confound their language, that they may not understand one another's speech. So the LORD scattered them abroad from thence upon the face of all the earth: and they left off to build the city. Therefore is the name of it called Babel; because the LORD did there confound the language of all the earth: and from thence did the LORD scatter them abroad upon the face of all the earth.

Perhaps attempting to build one character set for all languages is just another Babel-like act of hubris, and the security vulnerabilities are God's judgment upon us for not sticking with the mess of incompatible local encodings that preceded it.

Comment by failbuffer 3 days ago

I mostly just blame ancient Shang diviners... were it not for them, there would be no need to expand beyond a 16 bit fix width character set.

https://en.wikipedia.org/wiki/Oracle_bone_script

Comment by BadBadJellyBean 3 days ago

What is the alternative? ASCII isn't it because there isn't only English out there. All the different encodings we had before weren't it either.

Having only ASCII in urls is incredible US centric. Yes it brings problems, yes they are solvable. It takes work to not have US and it's writing system to be the center of everything.

Comment by cowlevel 3 days ago

A single language per file

Comment by duskwuff 3 days ago

... might sound like a reasonable solution until you start trying to write about languages. It is unreasonable for a sentence like "In Japanese, the word for 'hello' is こんにちは; in Korean, it's 안녕하세요." to be unrepresentable.

Comment by cowlevel 3 days ago

Then language change tags which draw a box around the different language.

Comment by colejohnson66 2 days ago

Why use that when a "simple" dictionary structure mapping codepoint to language exists? It's called Unicode.

Comment by JdeBP 3 days ago

We tried that in the 1980s with MS/PC/DR-DOS and code pages. We also had BBS networks where people shared such files around the world. One language per file turned out to be a bad idea.

Comment by cowlevel 3 days ago

To be fair, that was actually one language per computer.

Comment by JdeBP 5 hours ago

Oh, it really wasn't. If you want the terrible reality, start with how FidoNet CHARSET and CHRS 'kludge lines' tried to make this work.

Comment by octoberfranklin 3 days ago

UCS-2. What Javascript uses.

Comment by rerdavies 2 days ago

Javascript uses UTF-16, not UCS-2. An tragic historical mistake. I imagine they would use UTF-8 if they got a chance to do it all over again.

Comment by colejohnson66 2 days ago

It's because JavaScript was created in 1995. At that time, everyone believed UCS-2 was "all you'd ever need". Well, Westerners believed it. It's why Han Unification existed; to try to perpetuate that lie. It's why Windows is UTF-16 at its core.

Comment by BadBadJellyBean 1 day ago

Witching the default encoding is easy. Ask the python people. /s

Comment by hollow-moe 3 days ago

rnicrosoft.com sure didn't need unicode tricks to fool people lol

Comment by octoberfranklin 3 days ago

Pitching a solution to a problem they created. Yes, Unicode has become a racket.