In the course of putting together the encodings (called code points) in Unicode, a number of decisions had to be made regarding the current existing encodings, particularly well known and/or well established ones. In some cases, even though the Unicode Consortium has a particular policy regarding some encode vs render issues, there are inconsistent inclusions due to this grandfathering of prior established encodings and (to be quite honest) outright mistakes on the part of the Consortium. The question of precomposed characters versus combining characters is a classic one.

As a simple example, let’s take the greek letter alpha — ά — with an acute accent. Should this be represented as ᾅ or as ᾅ? (See here and here for the code charts.) In other words, should a single fixed code point be used to represent something that’s really a combination of a letter and an accent, or should there be a code point for the letter and a code point for a combining accent which is then combined with the previous letter?

My two cents worth is that since the concept of an accent applies to more than one character, it is an independent concept, and thus combining is the way to go. Not to mention more economical — with this approach, to add an accent to any other character requires only one additional code point in the charts. Implementing it the other way means for each character that might be accented, a second code point must be reserved — and as that cannot possibly be comprehensive, it will be by nature reserved to “legal” or “existing” combinations at the very least. So why are both present? That’s the grandfather clause at work, since the concept of combining characters postdates the establishment of many of the old encodings. But philosphically, the Unicode Consortium’s wise enough to agree with me

Now suppose a web developer needs to represent some manuscript online. The texts may themselves contain “mistakes” which are intended to be reproduced as is. For example, in (ancient) Greek, an epsilon will never be combined with a circumflex due to the rules of the language. But perhaps an idiosyncratic author did so anyway, or a manuscript is badly marked up, etc. If the only encoding at hand for ancient Greek contained only the “legal” stuff, I’d be out of luck if I needed to show this. Given this, clearly the use of combining diacritics makes every sense. While there is no precomposed character for epsilon-with-circumflex, I could still use ε͂ to represent this — ε͂̂ …

But wait! That’s not quite the end of the story! Not all browsers are good about combining diacritics. It’s actually something of an artform, and the positionings will depend whether the base letter is small or caps, and whether there are other diacritics also being combined with it. Frankly, most browsers don’t cope so well. So to get a usable display in as many cases as possible, I find it worth scanning ahead in the text to find all the associated diacritics and checking for the existence of a precomposed character before attempting to do it through building it up.

Test Case

Let’s take an interesting example here:
That is an alpha with a rough breathing, an acute accent, and an iota subscript.
If the alpha is uppercased, it looks like this:
Notice how the positioning of the iota subscript changes when alpha is capital, and both the breathing and the accent sidle a little to the side to get out of the base letter’s way.

I’m going to demonstrate how this pair of letters looks in precomposed versus combining notation in assorted browsers and operating systems. First of all here’s the table I used to build up my examples. The first column has the precomposed character for each of the above images. The second has the base character plus the combining diacritics for the above images. The diacritics are in “proper” order. This order is particular to each language, and in this case the order is supposed to be breathings-accents-iotas and that is what drives the order you see in the final representation — the breathing will always be to the left of an accent, and so on. The third column has the diacritics in reversed order. This is a very interesting situation that I don’t believe has yet been well enough assessed. I don’t even know that it should be: just as it’s reasonable to expect words to only make sense if their letters are ordered properly, it could be just as reasonable to expect diacritics to be listed in proper order. On the other hand, it might be reasonable for some programs (especially a word processor) to order them properly — these are computers and can churn out a few more CPU cycles, after all. I found it very interesting how badly handled this third column was, so I included it for interest’s sake.

There are several things to note here. First of all, particular diacritics, depending on the language of course, have particular positions. Some combine below, some along the top, some to either side and some as “overstrikes.” In addition, if there are multiple diacritics that are positioned similarly there is usually an order of precedence. Finally, if the letter is capitalised, that often affects the placement of the diacritic. In the images above (and in the table, if your browser is working correctly) the iota subscript moves to the side of the capital alpha.

First up on the chopping block: Internet Explorer (IE6 and IE7):

	IE6 (SP2)	IE7 (beta 3)
Windows XP SP2 w/Cardo
Windows XP SP2 w/regedits & w/Cardo
Vista (beta 2) w/Cardo	n/a

I will admit I was surprised at how well Internet Explorer did here when it’s usually so miserable at dealing with odd things, especially internationalization issues. However, do notice that it’s clear MS didn’t consider the case where the combining characters might not be “in order”. In the third column, the diacritics along the top haven’t been properly spaced from each other, nor are they shifted slightly in the capital letter version. And the iota fails to adjust to the capital letter.

Moving along to Firefox and Opera, we have the following table:

	Firefox (1.5)	Opera (9)
Windows XP SP2 w/Cardo
Windows XP SP2 w/regedits & w/Cardo
Vista (beta 2) w/Cardo
Linux (Ubuntu 6.06) w/Cardo
Mac 10.4.7 w/Cardo

What I find very interesting is how the registry edits for Windows XP improve both Firefox and Opera’s ability to display combining characters properly. However, this ability immediately disappears in the tricky third column.

I’m surprised by Opera’s complete failure to try combining the characters on the Mac, especially since it did combining in both Windows and Linux. Note that it simply listed each after the base letter in the order given in the table. Opera lets you define which font you want to use for the ‘Greek and Coptic’ and the ‘Greek Extended’ blocks. Now, I’ve set Cardo as my default font for everything, so that should override anything that’s there. “Should” being the operative word. Here’s the problem: I go to Preferences –> Web Pages –> Normal Font and I select Cardo there. However, this doesn’t seem to affect the display of Greek text in Mac Opera. This is odd, since it does affect the display of Greek in Linux and Windows Operas. I can instead change the display of Greek text in MacOpera by going through Preferences –> Advanced –> Fonts –> International Fonts and there selecting ‘Greek’ as the ‘writing system’. However, Cardo is not listed here. And since ‘Greek’ is actually Opera’s name for the ‘Greek and Coptic’ set, to make sure I can render all of Polytonic Greek correctly I also have to select ‘Extended Greek’ as the writing system. Cardo is not listed there either. Why is it not listed there when it’s clearly installed and is available system-wide? I do not know.

Let’s check the last two browsers of interest: Konqueror and Safari:

	Konqueror (3.5.2)	Safari (2.0.4)
Linux (Ubuntu 6.06) w/Cardo		n/a
Mac OS 10.4.7	n/a

I’m giving Konqueror bonus points for the creativity it shows in vertically stacking the rough breathing and accent. Unfortunately, it’s completely invalid on technical merits. The iota subscript isn’t even underneath the base letter, either. Konqueror needs to completely fix this aspect of their rendering engine. Surprisingly, though (and I say that because Konqueror and Safari both share the KHTML rendering engine) Safari comes out a winner here, rendering the combining characters correctly.

So what I see here is IE6 (shockingly enough), IE7, and Safari handling the combining characters the best. I say that because both of them are able to render the test cases properly without any outside modifications. Firefox’s correct rendering on Windows with regedits but not on any other operating system, nor on Windows without regedits leaves me to believe the credit lies not with the browser but with whatever effect the registry edits has on Windows. Still it did a decent job and it handled the reordered combining letters no worse than the properly ordered combining letters. Opera was neck to neck with Firefox for all the same reasons, but fell behind for its odd handling of the test cases on the Mac.

Konqueror and Safari wind up being the most puzzling since they use the same rendering engine. In theory, then, if one works, so should the other. I may try to rustle up other versions of Konqueror on other Linux distributions and if anything else might be going on.

Microsoft’s Registry Edits

Microsoft details up to three registry edits that are necessary to to set up Windows NT, 2K and XP for unicode functionality. (Vista comes already set up, and does seem to actually work.) The first edit is the most important. It enables Uniscribe support which is what Windows applications use in order to be able to render Unicode characters. It’s responsible for rendering input text, for substituting character variations according to context, and for ordering displayed text based on text flow direction.

The second registry edit adds support for the supplementary plane characters in IE (which I will cover in the next post). The third adds the ability (in Win XP only) to specify a default font for supplementary plane characters.

Presumably the first edit is what enables Firefox and Opera to be able to handle combining diacritics. I’m at a loss to explain IE6’s behavior, unless a fairly recent patch to it enables the same thing. Unfortunately, I don’t have an earlier unpatched IE6 to test that theory out with (if I’m correct, such an earlier version of IE6 would need this registry edit to display properly). Installing certain language packages will also install the Uniscribe module “behind the scenes,” but as I had the same language packs before and after adding the registry edits, that doesn’t explain this either.

The full instructions for doing the registry edits may be found here.

Excellent Resources

For much more informative commentary, check these pages out:

• Benefits of the Unicode™ Character Standard
• Character Sets And Code Pages At The Push Of A Button
• More about the character concept (the entire article is excellent as well)
• Combining Diacritical Marks

In the beginning was ASCII, at seven bits. And it was good, until someone noticed a few missing characters. In this way, ASCII with eight bits was born. But alas! There were even more characters to be respresented. And thus began the exodus in search of ways to show these missing characters.

And the way split and conmingled between encoding and rendering. After all, since no encoding existed for some characters, those attempting to render them on the screen made up their own encodings as they went. And things continued in this sorry mess for years, until Unicode was born…

Of course at this point, now various different programs such as mailers, forums, word processors, internet browsers and even programming languages all are in the process of being updated to understand unicode. It’s a giant bear of a mess along many fronts, although I do believe Unicode is the way forward, the essential problem is that it’s been introduced at this point instead of at, say, the point ASCII originally came on the scene. But no matter, things will sort themselves out. Over the years and years, but hey.

So what, exactly, am I talking about? In order to represent the alphabet, two things are needed. First is a kind of representation that says which character we’re talking about. So for example (decimal) 65 is used to represent the letter ‘a’. The second kind of thing is whatever is used to render ‘a’. It could be a, or a or a. These two things are the encoding and the rendering of a given character. Conceptually these two properties of a character should be distinct. In practice, of course, it’s not always been that cleanly handled, and there are some issues where the lines are legitimately blurred.

The problem was, of course, that the original encoding table (ASCII) was much too limited to handle languages other than English. To address this, a number of ISO 8859’s were developed to cover additional characters such as ß or ñ and other marks and symbols such as © and £. However, since rendering (or typography) was not considered in these representations. a number of legacy fonts developed that used additional proprietary (and conflicting) encodings for additional information not covered in the standards. And in all of this, languages that did not even use the Latin alphabet (such as Greek, Russian, Arabic, Hebrew, Japanese, Chinese, and so on and on) were definitely ill-represented overall. In most cases there are several possible representations and standards to use, which results in a nightmare for anyone trying to represent extended or other character sets in programs that make use of them. (Which basically potentially includes any program which ever tries to communicate with its user in anything other than international symbols, but I digress.)

Unicode

The idea behind Unicode is to create one giant encoding standard for all of this (leaving the typography alone and up to whatever rendering a particular program wishes to use, or whatever font set the user has installed). Sounds simple enough although even this idea is fraught with complexity and inconsistencies. For example Unicode absorbed many of the original encoding standards in order to ensure backward compatibility; and has made various inconsistent decisions on the inclusion of other characters in different ways. However. the underlying concept is sound, and if it takes another twenty years to refine it, the end result should still be better than the cacaphony there is now.

Enough with the background. I want to discuss most of this in the context of browser rendition, since this is a good deal of what I work on anyhow.

Let’s take a quick look at this:

I have it up here as an image to guarantee that everyone here can see this word. (I should note, by the way, that I picked this word out at random, but I do not know ancient Greek. When I looked it up, it turns out it’s an adjective meaning “eager to be of service”, which I find vastly amusing.)

In Unicode, I could encode this as follows:
πρόθυμος
I am using Numeric Character References here, instead of raw UTF-8, mostly because IE6 expects such representation although in theory either should be acceptable and certainly Firefox, Safari, and Opera are all happy with either. The codes can be found here and here.

In any case the above should render similar to the image above:
πρόθυμος (NCR)
I could also put it down like this:
πρόθυμος (direct UTF-8)
but Internet Explorer may not render this correctly.

Now if it did not, there are several considerations to check.

The first will be whether or not the page itself is correctly set up for viewing Unicode. Since I’m the one running this show, of course it is. There needs to be a declaration at the top something like this:

<meta 
    http-equiv="Content-Type" 
    content="text/html; charset=UTF-8" 
/>

Since I can eliminate that as a problem off the bat, the next consideration is the browser being used to view this. All the modern browsers (meaning: Firefox (all OS), Opera (all OS), Safari, Internet Explorer 6 (latest patches), Netscape 7.2 plus and derivatives) should understand this. Some of the older ones may not. In particular IE on the Mac never contained Unicode support. In many cases, setting several menu options may be necessary to enable the support — check the documentation for the particular browser for “character encoding”. For IE, check the “user-defined” options. View->Encoding needs to be set to user-defined, and then in the Internet Options, a suitable font needs to be selected for the “user-defined” font (eg, not the Latin, etc fonts).

Different browsers will support unicode at different levels. For example, on IE6, it’s not only necessary to enable the character encoding support, but also to do certain registry edits for both the browser and the operating system (IE7 appears to contain more support for Unicode, fortunately). Firefox and Opera only need to be informed of a compatible font. Safari actually breaks down the “unicode” fonts into the different regions on the (very reasonable) assumption that one might use different unicode fonts for different languages and not some “universal” unicode font. So in Safari, setting the unicode “Greek” to the correct font will allow the above to display.

A good font for the above display that I would recommend the Cardo font. There are several out there, including Arial Unicode MS, but I do not recommend this or any of the older ones, as support is only to version 2.0 of the Unicode standard, and we’re well past that now. I’ll return to this point later on. For now, I’m going to detour briefly into legacy fonts, just to illustrate why they were such a bad idea.

Legacy Fonts

I’m not even going to render them on this page . Actually, I cannot render them in this page, because they require different meta declarations, i.e.:

<META http-equiv="Content-Type" 
          content="text/css; charset=ISO-8859-1"
/>

In any case, the representations for the above word in these fonts, respectively, are

<font face="Greek Old Face 98">prÒqumoj</font>
<font face="Athenian">prñyumow</font>
<font face="Ismini">pr¿uymoq</font>

Note the need to explicitly switch font face in order to obtain the results I want. Unless the entire page will be rendered in one of these fonts, I have to switch around as needed. Unicode simplifies all of this, and theoretically given sufficient support (a unicode keyboard, unicode display in the editing software, etc) I would not even need to use the numerical character representation of specific characters. However, as I cannot guarantee all visitors to this site would be similarly equipped, this is a fallback for now (and indeed, since I — nor anyone — will never have a keyboard that includes all the possible alphabets and symbols for non-alphabet languages, this kind of workaround will always be needed anyway).

Interestingly, some of the newer browsers themselves have bugs in attempting to handle legacy fonts. For example Firefox had trouble with the style font-family method of switching between fonts, but was able to display correctly once it was changed to inline font face notation.
So

<font face="Greek Old Face 98">prÒqumoj</font>

Minor note on WordPress

This is getting lengthy and I still have more stuff I want to discuss, so I’m going to wrap up here and finish the rest in the next post. But I thought I’d point out that somewhere along the line in the course of editing and re-editing this post, the NCR’s I was using to illustrate the various unicode words and such above kept getting rendered into their UTF-8 representations. I wanted NCR’s for several reasons, partly because it allows me to reference the Unicode Manual on which ones they are, but also because some browsers require NCR. But somehow, and I’m not sure if it was internally in WordPress or some combination of cut and paste in either Windows XP or Ubuntu that did it, but it’s a classic example of the annoyances that are still present when dealing with “exotic” character sets.