Unicode in Action

From Pat Smith and Jeffrey Jia, who've been working on the Psiphon project:

One of the more challenging parts of getting Psiphon up and running was the internationalization and localization of the interface. The problem was Unicode. As we discovered, Unicode (or rather, pretending it does not exist) is a big problem in software. In order to ensure that Psiphon would work with "all" languages, we needed to get a language such as Chinese or Persian working. Both Chinese and Persian use the UTF-8 encoding, and they do not use the ASCII character set we are used to dealing with. Chinese and Persian were two ideal languages to try because Chinese requires more bites to be displayed, and Persian is also a "right-to-left" language.

The first big problem was that of the development tools that we were using to develop Psiphon, not one was able to properly handle the Unicode characters. During initial tests on the Persian translation file we were unable to tell if the problem was with our data file, or with wxPython and PythonCard. After four to five hours of working along side Michelle Levesque, comparing how our different tools handled the Unicode, we determined that it was our development tools that were causing the problem.

Of Komodo, Cygwin, Windows Command Prompt, and IDLE, it appeared (at first) that IDLE was the best at handling Unicode. Komodo could probably display the data file when it was opened, but its Python shell interpreter would print gibberish when it attempted to handle the data. We determined it was a codec problem with Komodo's Python shell, by observing patterns in the gibberish that was returned and using Google to see if the results were a common occurance when dealing with Unicode. We had moderate success with IDLE, however, after further tests with more Unicode, it seemed that IDLE also had problems dealing with alot of Unicode. We were not able to find any tool that was capable of handling all Unicode correctly, so we decided to continue working on the internationalized interface without a satisfactory tool.

We soon realized that the Unicode was not being displayed properly because a non-Unicode build of wxPython was installed. After uninstalling it and installing the correct Unicode build of wxPython, the problem was still not fixed. wxPython includes a demo application which includes a Unicode example, with the Unicode build installed, this demo should run without a problem, this was not the case. After many uninstalls/reinstalls, it was discovered that there was an old partial install on the path that was part of an unsuccessful Unix install under Cygwin. Removing that allowed the wxPython demo to run smoothly, but Psiphon's Persian translation was still causing the program to crash.

Reading the error trace we were able to determine that the problem was now within PythonCard's handling of Unicode. The problem was traced to PythonCard's multicolumnlist module. A multi-column list is used in Psiphon to track the connected users, and the headers of each column require internationalization. By reading through the multicolumnlist source code, and reading the Python 2.3 documentation for Types, we found that PythonCard was using StringType, which according to Python 2.3 does not include Unicode strings. The PythonCard source would produce an error if Unicode was given to a multi-column list. By changing all occurances of StringType with StringTypes (includes Unicode), we were able to avoid the multi-column list error. However, it does require users to change their own version of multicolumnlist.py if they choose to run Psiphon from the source. We reported the bug to the PythonCard author Kevin Altis, and he posted on the PythonCard mailing list that the bug will be fixed on the next release.

Despite all this, we were still getting errors when Psiphon tried to display Unicode. Since we were sure (well, pretty sure) that the Unicode wxPython wasn't the problem, we determined that the PythonCard resource file was the problem. This lead to many hours of considering the possibility of redoing the interface entirely in wxPython, which would have resulted in many hours of work to migrate the existing GUI away from PythonCard and into just wxPython.

Back to Kevin Altis, who informed us of a bug: Unicode was handled in resource files in the same way it was handled in the multi-column list component. Unfortunately, the suggested fix only worked if the resource file was being written in Unicode, and not dynamically creating a resource file while a program is running. Robin Dunn of wxPython gave much appreciated help in examining the resource file generating code, and with his help we were able to get Psiphon working with Unicode languages. However, we were unable to get the "right-to-left" working with Persian. Since we needed Psiphon to be multi-platform we could not simply use the GTK wxPython build. GTK itself is able to handle "right-to-left" easily, but wxPython does not have its own related options.

The Unicode problem took a lot of hours to fix, but forced us to understand Unicode, and why we should never write a program which does not handle Unicode properly. We also gained experience in sorting out problems in open source software. Tracking down a new bug in an existing tool gave more insight into how the tool itself works. It also taught us how to better debug code, and to realize that sometimes the bug may not be in the code you are writing, but rather in the code from a tool that you are using. Overall, the time invested in fixing the Unicode problem was worth it for the knowledge that was gained.