Unicode — Supplementary Planes

Back to Unicode. Even Unicode needs room to expand. The Basic Multilingual Plane (BMP, also known as Plane 0) has codepoints 0 through FFFF hex (0 through 65,535) and contains, ignoring various special mathematical characters, historical quirks, and just plain oddities, most of the current alphabets and symbols in use today, even including the vast array of Chinese kanji. But of course there’s more room needed once historical alphabets and other special character sets are also considered. Supplementary planes such as Plane 1 are simply sets of 65,535 code points following Plane 0 (some of us refer to these as astral planes ). Using hexadecimal notation, it’s easy to spot Plane 1: all Plane 0 codepoints use four hexadecimal digits; all Plane 1 use five, etc.

Here is an example of a Plane 1 character: 𝔊 which renders as 𝔊 (marks a septaugint reference). In case anyone viewing this is having trouble, here’s an image:

To set things up to view this can be an interesting process. First of all it’s necessary to find a unicode font that supports plane 1 characters. Not all such will actually show the above, for example Code2001, an otherwise excellent unicode font, does not include historical Greek musical notation however Cardo works nicely. Just because a font is unicode based does not mean it will be “complete” (such a font would be staggeringly large). In particular, when choosing among unicode fonts, pay attention to which version of unicode is being supported, and what the target languages are. Plane 1 characters start to appear in Unicode versions 3.2 and up.

Second, the operating system might need slight adjusting to see Plane 1. I’m talking, of course, about Windows, XP and earlier versions. This page discusses how to do the registry edits necessary both for the operating system and for IE6 .

Linux/*nix, Mac, and Vista are all presently able to handle Plane 1 without modifications. My understanding is that for Windows Me and anything prior to NT, it’s hopeless.

Third, applications in general and browsers in particular may need slight tweaking to see Plane 1 characters. As usual, IE6 is the worst offender in this regard requiring not just setup but also a registry edit. In general, once the browser is set up with a Plane 1 aware font, it’s good to go. Firefox, Safari, Opera, and Konqueror all fall under this category.

A few notes: IE6 needs to be set to user-defined encoding plus the extended font needs to be listed under User Defined (rather than any of the specific regions listed, which refer back to non-unicode encodings as discussed in part 1). Opera actually reverses from the general MO of other browsers: under Tools->Preferences->Advanced->International Fonts, it’s possible to set a particular unicode font to a particular language (going by Unicode’s code blocks). This is the direction other browsers should no doubt take in the future.

Overview

So in Windows and Mac, there’s this so-called regional settings that only “allows” you to change your regional settings five times before “locking” it in, courtesy of the entertainment industry determined to dictate every choice to their customers and squeeze every penny out of them into the bargain. Ah, the fresh smell of capitalism in the morning.

The way it works is that DVDs which are “Region 1″ (2, 3, etc) are CSS (content scramble system) encrypted. The settings on the computer indicate which “region” it is, and that will refuse to play those DVDs not matching the set region. This is a result of collusion between OS companies (eg, Microsoft, Apple), DVD drive companies (Panasonic, Sony, etc), and assorted software programs. It’s an ongoing legal tangle, and also when OpenSource is tossed in, gets interesting fast, as OpenSource will not generally “respect” the encryption and simply play DVDs regardless, although such a user generally will need to install a few extra programs and such to make it all work so in the end, everyone winds up having to work around this issue.

Oddly enough, it’s almost simpler when discussing DVD players. For these, just google up the brand name of the player, along with the terms ‘region free” etc, and it’s possible to find, for most players, the codes necessary to convert the player into multiregional or region free players. Once that’s done and it’s hooked up to the TV set, playing any DVD is not a problem.

Approaches

But back to DVD drives in computers: there are several ways around regional restrictions. One has been to hack the firmware on the DVD drive. This isn’t an especially good idea since such hacking can render the drive dead a certain percentage of time. So I’d toss that out.

A second option is to copy the DVD first, stripping out the CSS, and burning the altered copy to a blank disk and using that DVD instead. This works, but is a little cumbersome — having to copy each encrypted DVD before viewing it.

Finally, there are software programs that simply bypass the operating system restrictions. These can be built into the dvd players themselves, or be a separate program functioning on its own, allowing the user to use whichever favorite software for viewing the disc’s contents.

Windows

For Windows users, most of the options center around the third one, although a number include the ability to use the second approach as well. For example DVD Region + CSS Free and DVD Region Free Master (both very reasonably priced shareware). The programs work at the software level to bypass any restriction on the computer’s dvd drive, whether or not it has been locked (changed more than five times) and without altering its allowable changes.

DVD Shrink allows users to copy commercial dvds to blanks. The program allows for copying and compressing and/or eliminating parts of the original dvd, including stripping the CSS encryption.

Mac

For Mac users, the picture is a little different. There are some programs out there but there don’t seem to be that many (and which seem to be for older versions), which puzzles me because OSX is Linux based, more or less. Some of the issues seem to revolve around which Mac one has, and which dvd drivers it has. VLC may work for some Mac users, but does not work for all. I did find this article which summed up the issues for mac users and described one solution the author worked out. I have also seen speculation that Mac’s new Bootcamp software might allow Mac users to run the Windows region free software + dvd playback to obtain similar results (but I did not find anything written up by anyone who’d tried it to see).

There’s a program called MacTheRipper which allows copying of DVD’s minus the encryption. Also check out DVDBackup to back the DVD up on the computer.

Of course with the new Intel chips, a dual boot WinXP and mac, or dual boot with Linux and Mac and move back and forth as needed.

Linux

As an Open Source operating system, Linux’s difficulties with CSS encryption stem from the proprietary nature of the software. In most cases, what a Linux user needs to do is download the additional software and codecs to view DVD’s. Most distributions will have instructions available somewhere on how to do this: googling on the distribution name and region free will in most cases uncover instructions specific to the distribution. For example, googling on “ubuntu dvd region free” (without the quotes) will uncover tutorials such as this.

There are a wide variety of players, such as mplayer, Xine, Kaffeine, Ogle, and Totem. Most distributions will come with one or more of these, the trick is enabling the play for encrypted DVDs as covered in the previous paragraph. Of these, Kaffeine is generally considered the best of the lot. The rest may not have a full gui interface, or sync properly, be somewhat buggy, or have other limitations.

As for creating region free DVD’s: one method is outlined here and another nice overview is here. The basic idea is as with the mac: create a copy or backup on the computer, and then burn selected portions to the blank disc.

Cross Platform

One name that kept popping up was VLC which has versions for Linux, Windows, and Mac. I have not tried this myself, but the user feedback/ratings look pretty good.

Further Information

Check wiki for more overall information on DVDs, DVD software, and dealing with CSS.
For complete informatin overload (but well written and easy to skim for specific questions), check out the DVD faq.

Legal Issues

Notes about legality. It’s becoming a bit more common to find notices such as this one even outside the U.S. Not only do laws vary from country to country but they appear to be in the process of changing, as more attention is paid to this issue. I should stress that this article is provided for informational purposes.

Also, I am not a lawyer, but this is my view: When I purchase a book, I am entitled to read it, whenever and whereever I want to. I can read it as many times as I want to. I can do anything to the book itself — write in it, tear pages out, laminate it, store it, throw it away, give it away, or sell it. I may not make copies of it, with the exception of small excerpts and such under “Fair Use” doctrine (eg, quoting passages in a review, that kind of thing). Extending that concept to a DVD, I personally find the encryption and regional restrictions problematic, as these may prevent me from enjoying something I have paid for the right to view. I therefore view getting around the encryption to be a legitimate exercise in being able to make rightful use of a product I’ve paid for.

The gray area is, obviously, when DVD’s are copied. And here it’s unclear: for example it’s been established as acceptable for users to make personal copies of audio tapes or CD’s — for example making a cassette tape for personal use, because one’s car does not have a CD player, or “timeshifting” where one makes a VHS copy of a broadcast to view later. No reselling is involved. So making a copy of a DVD for personal viewing ought to fall into similar categories (watch a DVD on laptop), but the companies are nervous because at this point, redistribution of copies is ridiculously easy over the Internet (not as easy to make VHS tape copies and distribute, nor audio cassette copies and distribute). That, in a nutshell, is why DVD copying is an “issue” and why the conventions are inconsistent between book, tape, and CD/DVD.

Came across this article the other day and I’ve been thinking about it. Setting aside his somewhat confusing terminology (”Free Software” really should be called open source, because the very program he is railing against, Picasa, is free software. The source code is proprietary, but the software is still free…), he nevertheless touches on a number of issues I’ve been fascinated by for the last several years or so.

In any case, like any good geek, I’ve used freeware for years. Who wouldn’t? And of course I’ve always been generally aware of open source software, as anyone on Unix would be, through the efforts of gnu, linux and the like. However, since I was using Windows increasingly in my current job and lazily on my home computer, it wasn’t until about two years ago when I decided I was going to migrate to Linux and then last year when I completely removed the Windows installation that I was looking much more closely at Open Source.

We already know that the Open Source model is sound. There’s more than two decades worth of solid work, starting with the early GNU projects to today’s Linux distros and more to show that. What’s more, especially for the more popular and widely used programs, the churn rate on bug fixes and feature additions is much higher. You only have to look at Firefox’s quick turn around on bug fixes, and the number of releases its made as compared to Microsoft’s completely anemic Internet Explorer 6 which gets only a minor facelift some seven years later with the release of IE7.

But if you’re not compiling things, if you’re not poking through the code, if you’re happy using just executables, then the philosophical differences between freeware and open source aren’t going to be obvious. And certainly there’s quite a bit of other freeware that enjoy the same pace of responsiveness and bugfix/feature adding. Linux users, even while preaching the open source mantra, tend to rave as much about google software as anyone else, and last I checked, googleware, while free, is most certainly not open source.

There’s a slew of other considerations as well. Open source software is more likely than most to be present on multiple platforms. A good piece of software whose code is accessible can be ported by someone interested in it elsewhere. This simply doesn’t happen with proprietary code; only the owner of the code will make any necessary investment to do so, and many don’t. And the beauty of using the same program whether you’re on Windows, Mac, or Unix is that you don’t have to memorize new interfaces. You’re more likely to be able to interact without issues with someone on another computer. Send them a file and it will be correctly interpreted and opened. The frustration factor goes down and the usability factor goes up.

However, I think Picasa is a slightly different issue. The only reason Picasa is available on Linux is because it’s bundled with Wine, an open source program that allows Linux users to run Windows applications. Google made extensive fixes and extensions to Wine so that it would be able to run Picasa in Linux. We therefore have a piece of open source software which has been extensively upgraded in order for Picasa to make its Linux debut.

This actually takes the entire Picasa debate out of the usual free vs open domain, and indirectly brings me to another issue especially relevant with today’s issues on domestic surveillance by the governments: the final reason that pushed me over the edge into getting out of Windows was the whole DRM and privacy concerns. This involves not just the DVD protection stuff but also the trusted chip technology (blog of helios has a pretty good run down on this whole issue worth reading — check also for which hardware is involved). You’ll notice that now these trusted chip sets are going to be on Macintosh computers as well, with their well publicized move to Intel chips.

Linux is literally the only way out of this for the foreseable future, and it’s Open Source that will guarantee that. Even if DRM were somehow put into it, anyone could recompile without it. Folks outside the country could keep versions available.

And how did I come to this digression with Picasa? Back to Wine: it could well be a way for die-hard Windows users to get away from the DRM technology while still running the programs that they know and love. I just love the multiple levels of irony here.

It’s been a few days since the official release of the Dapper Drake Ubuntu 6.06 distribution. It was actually not a huge change for me, because I had installed the Beta distribution six weeks ago and so got the upgrades in increments (sometimes, it seemed, every single night). Overall I had no glitches, save one in which my NetworkManager installation got uninstalled in the initial switch to the Beta distribution.

In any case, the transition went very smoothly for me. I’ll be honest, though. I’m a fairly lightweight user and I have an aging laptop (only in computer land is a two year old product “ancient”) so I did not see performance improvements — or degradations for that matter — during beta and final installationst. I use a locally run USB printer, and my wireless works although I’ve gone back to the command line iwconfig for now. I also haven’t actually gone in and double checked what the upgrade did to my LAMP installation. I should do that…

But for others the upgrade did not go so well, and it’s been enlightening, to say the least, to read the various discussions. For a bit of background, it’s important to know that Ubuntu initially started out by promising six month distribution releases. An important feature of a linux distribution is its timely release. In open source, the pace of upgrades, security plugging and bug fixes is high. A distribution not only collects a particular set of linux tools but also integrates particular versions and fixes together which can be nontrivial. Not every linux user wants to fiddle with package dependencies of which there can be hundreds. In addition with regular distribution releases, other projects and business can plan their own releases around the distribution.

So, when Ubuntu announced a six week delay with the Dapper Drake release, it caused some consternation but inasmuch as this was an intended LTS release, most seemed to settle on “wait and see” on whether the delay was justified.

Problems have surfaced quickly, though, and Ubuntu will need to resolve them given the double barrell of having delayed this particular release and of its being the first intended for long term support. The problems include the following:

1. Issues between CUPS, Gnome, and certain printers as documented here. What’s very interesting to note is if you scroll/search down the comments, look for Pascal’s comment, a little more than half way through (searching on the term “Project Ridley” in Pascal’s comment should work) where he says “The problems described here are because gnome-print libraries are not maintained and buggy. […] Until then we are stuck with gnome-print in dapper (GTK+ 2.8). ” However, many of the comments describe how the printers worked in 5.10, and then partially throughout the beta Dapper releases until they stop working altogether in the final release, so it still appears the issue could be addressed from wthin Dapper until these libraries are upgraded. Even if it’s not Ubuntu’s fault, the point is that people can’t use the printers in this distribution.
2. A live CD that doesn’t boot is downright embarrassing.
3. More troubling are assorted wireless issues, here, and here. In particular check the dates through the comments on the first bug report. Reading through them, it’s clear that this was discussed for three months.

Most of these do appear to be external library and hardware issues, but they were not present in Breezy (5.10). That makes this distro appear a step backwards for those users hit by these bugs. It also raises the question of how Dapper could work around these issues anyway since the systems experiencing these problems are the same ones running 5.10 (which further undermines the appearance of these bugs being hardware related). I’m very curious about what exactly happened in the last set of releases to expose the bugs from the penultimate beta Dapper to the final Dapper.

Now there are several issues that Linux gets to deal with that Windows doesn’t, at least from the user perspective. Most people purchase their computers with Windows already loaded (and tuned for their hardware — think of all those extra CD’s you get from Dell/Gateway/Compaq/etc with the drivers, etc) so they don’t see problems. And if they just purchase another computer when done with this, they may never notice. But, as practically anyone who has reinstalled Windows can attest, you can not simply reformat the hard drive, pop that CD back in and go your merry way. You often have to go searching for the latest drivers and such to install (and if you were unlucky enough to have your internet access disabled along with the rest, you’ll be searching on someone else’s computer and transporting the results back on your flash drive…)

And of course you can’t readily purchase a computer with Linux installed on it for you. I’ve heard it’s theoretically possible. For example supposedly if you request it of Gateway, they’ll do it, although that was a few years back and of course I have no idea what distribution it would be. And of course, given that there’s hundreds of distributions available, the chances of your favorite Linux being available preinstalled on a computer you plan to buy is remote. But I don’t see this as a bad thing, so much. It forces the Linux distributions to constantly improve how they self-install. Windows doesn’t have to worry about this; most of their installations are done by techs before the computers are shipped out. And over the years of lack of competition, they’ve never been forced to improve that process.

Anyway, the one thing I’ve been searching for and not finding is any announcement of patches or releases to fix or address these problems. I’ve had a few upgrades come through since the Dapper final release, but these appeared to be relatively minor fixes. Hope these are addreesed in some official way soon.