Man, this was a great post. the explanation of canonical value, the heat map and the unicode codespace chart were my favorite.

I believe «U+0041 “a” latin capital letter a» should read «U+0041 “A” latin capital letter A».

Thanks; fixed.

I've been working with Unicode for years and I've learned many things from this post. Thanks!

Really well written and comprehensive post! Thank you so much! Finally I understood Normalization on unicode! haha

Really informative and well written, thank you.

Thanks a lot... now stuff finally makes sense.

Small point, this this is incorrect. I saw the same error in Microsoft Technet around 16 years ago, and complained, but I don't know if it ever got changed. "In UTF-8, each code point is stored using 1 to 4 bytes, based on its index value." Apparently there is an RFC that limits the code pages, so only 4 bytes are needed at the mo, but in the original spec, it's 6 bytes. http://stackoverflow.com/qu... I just checked wikipedia, and it's wrong there as well.

It's not "wrong". You're only allowed to use up to 4 bytes according to the current standard. (See section 3.9, definition D92 in the Unicode Standard, for instance.)

It's true that UTF-8 could easily be extended to allow more bytes, if the codespace was ever expanded again.

yeah I read up on it afterwards. I originally read the spec in 1999 pre-the change to reduce the max number of bytes. no idea why they did that. Basically UTF-32 allowed more or less 2^32 values, but in order to convert this to bytes, you needed up to 6 bytes due to the need to signal an extension. So the "improvement" was to reduce the allowed values. Since this requires more logic, seems a bit silly to me, it was fine as it was, would take no more space unless some new character sets were developed or we encountered Aliens. AFAIK ISO/IEC 10646 UCS is still 6 bytes.

UTF-8 was never defined to use 6 bytes. At some point, it was discovered that some implementations would map a supplementary-plane code point to a surrogate pair and then map the surrogate-pair code points to UTF-8, which would result in a six-byte sequence and also create a security attack vector. So definitions were tightened to clarify that you must not use a six-byte representation.

Well I certainly remember reading that it was defined to use 6 bytes in the original specification, and the accepted answer on stackoverflow mentions the same: http://stackoverflow.com/qu....
I did post a message on Microsoft TechNet in 1999 pointing out that TechNet said 4 characters while at the time the spec said 6 characters. (with URLs for the sources)
It hardly matters in any case. I've learned that nowadays UTF-8 characters can be a maximum of 4 bytes. Apologies that my knowledge wasn't up to date.

The original specification of UTF-8, captured in RFC 2279, did allow up to 6 bytes to represent code points up to 0x7FFFFFFF, the original scope of ISO 10646. This had nothing to do with encoding surrogate pairs directly in UTF-8, which is the evil CESU-8.

You're right that RFC 2279 and even ISO/IEC 10646 itself (until the second edition of the unified 10646 standard) defined UTF-8 to allow six-byte sequences.

But I was assuming the context here, which is The Unicode Standard, and that never defined UTF-8 with six-byte sequences.

As for Microsoft TechNet documentation, I very much suspect it would have been describing Microsoft support for Unicode, not ISO/IEC 10646.

I'd seen all these combining marks and assumed there had to be some system for canonical representations, but knowing the details is really interesting. Thank you very much for this article.

One other thing I'm curious about is that there are a number of math symbols that are have 'half' or even 'one-third' versions. For example:
⌠
⎮ ⬅︎ That appears meant to be stacked to form one large ∫ symbol.
⌡

And the naming 'TOP HALF INTEGRAL' 'INTEGRAL EXTENSION' and 'BOTTOM HALF INTEGRAL' seem to indicate that the middle one may occur an arbitrary number of times. If combined properly, it seems like they would all be part of one 'grapheme', but I can't think of a sane canonicalization or counting scheme for that.

Why did the Unicode consortium decide to do this? How is it supposed to be counted or searched for if someone has a stacked up integral with, for example 5 parts? The decision to include this seems primarily motivated by display purposes. And that seems to sort of be at odds with what Unicode is about.

Hi Omnifarious, I think those are legacy characters from DOS "extended ASCII" (code page 437). It has a bunch of characters for drawing lines, boxes, and stuff in terminals. And also integrals! You line up the characters in adjacent rows of the 80×25 terminal grid and they visually connect.

I don't think these characters have any special grapheme semantics, or any role in proper math typesetting. I wouldn't expect to see them used outside the terminal context.

The terminal or a printer. Old printers tended to have a one-directional printing mode, in which printing was only done when moving the head from left to right (so twice as slow), so table borders and special symbols like the above aligned properly.

Hey, Nathan, a good write-up. But here's an error:

"UTF-16 is also used by the Microsoft Win32 APIs; though Win32 supports either 8-bit or 16-bit strings, the 8-bit version unaccountably still doesn’t support UTF-8—only legacy code-page encodings, like ANSI."

The code page ID for UTF-8 is 65001 --- see here:
https://msdn.microsoft.com/...
In the Windows SDK, this is defined in winnls.h.

PeterCon [MSFT]

Hi Peter, sorry for the late reply. As far as I can tell, CP_UTF8 can only be used in conversion functions like MultiByteToWideChar, and there's no way to make the "A" versions of Win32 API calls (like CreateWindowA) take UTF-8. That's what I meant by that sentence.

the life of a programmer, everything looks like magic to me ...hahahaha

UNICODE MAP FULL INDEX HEATMAP Usage Frequency.
https://www.ydady.eu/articl...

https://uploads.disquscdn.c...

Even though I've been working with unicode for years, through your post I finally comprehend the things I had developed an intuitive understanding for. Thanks!

Andrej Karpathy sent me here :-)

Me too! Great post. Learnt something useful.

came here only 7 years later... thanks for the excellent piece!