Safe from the Losing Fight

One of today’s fads in software engineering is supporting multiple languages. It used to be that each language or script had its own code point system (or encoding), with each code point representing a different character. For reasons of convenience, the various scripts were incompatible, they could not identified by simply looking at the code points, and an identifier describing which script the text was in was not allow in the same zip code. Sometimes this caused problems with engineers who had weak constitutions; was that ‘c’ or a ‘¥’? Experienced programmers knew the correct answer was to cycle through all the known scripts, interpreting the text with each in turn, and ask the user to tell them when they could read it or saw the hidden picture. These were the earliest known captchas.

The Unicode Consortium was unhappy with this because they were not the cause of the mass confusion, as a result of being late to the party. They devised a scheme in which each character had its own unique code point. They also allocated enough code points to represent all the characters of a lot of different languages. They even added a unique byte sequence at the start of any Unicode text to mark it as Unicode. And thus, all was well and good as long as you didn’t mind having text that took four times more space than usual, and wasted three out of four bytes. The Unicode Consortium at first wasn’t interested in fixing this problem until they realized they could use it to add more “features” (read: confusion). The Consortium begat UTF-8 and UTF-16 in order to fill this need. UTF-8 encoding allowed most characters to be encoded in 8 bits, with the rest as escape sequences, and UTF-16 allowed most characters to be encoded in 16 bits.

Originally people implemented these types in C by using unsigned char (UTF-8), unsigned short (UTF-16), or unsigned int (UTF-32). At the time of adoption of Unicode both Win32 and the Mac Toolbox used UTF-16. It was a nice tradeoff between size and efficiency. For most characters they were only wasting one byte (as opposed to three bytes in UTF-32), but could still assume most characters where just 16-bits (as opposed to UTF-8 which escaped anything not ASCII). Life was good.

As most standards committees, the C/C++ standards committee were bent on death and destruction. They saw people were using this newfangled Unicode, and that it was almost sufficiently confusing. The standards committee wanted to advocate this confusion while adding even more. To achieve their demented objective, they introduced wchar_t and std::wstring. But which encoding of Unicode did it use: UTF-16 or UTF-32? BUHAHAHAHA! In their greatest show of leadership to date, the standards committee refused to say. It would be a surprise, and they would hate to spoil a surprise. wchar_t was defined to be more than a byte but no larger than jet liner.

With this new edict in hand, compiler and library writers quickly got to work. Instead of following each other’s lead, they each implemented wchar_t and its supporting libraries as they saw fit. Some saw the benefit of making wchar_t UTF-16. Others wanted it to be UTF-32. And thus, the standards committee bode their time.

Since both Windows and Mac OS (Classic) had adopted UTF-16 already, the compiler makers implemented wchar_t as UTF-16. But this was just a trap, meant to ensnare hard working cross platform engineers. Engineers who worked on software that ran on Windows and MacOS started using wchar_t. It was easy and worked well. A little too well.

Meanwhile, Unix vendors had decided that wasting one byte was insufficient, and that wasting three bytes per character was definitely the way to go. Besides its not like anyone on Unix was using Unicode for anything other than Klingon.

The trap was sprung in 1996 when Apple purchased NeXT and its Unix based operating system. Like all good traps no one realized what had happened for several more years. It wouldn’t be until 2001 when Mac OS X was released and Steve Jobs started after developers with cattle prods to get them to port to Mac OS X. Unfortunately for the standards committee, some developers continued to use the old developer tools, like CodeWarrior, and old executable formats, like CFM/PEF, that implemented wchar_t as UTF-16. But the standards committee was patient. They knew they would prevail in the end.

Apple would turn out to be the instrument of the standards committee. They continued to improve Xcode until it was good enough to actually build most of their own sample code. At the same time, Metrowerks finally won its game of Russian Roulette, and stopped development of CodeWarrior. Apple delivered the final blow when they announced they were moving to the Intel architecture and that they had the only compiler that supported it. A compiler with a secret.

There were screams of anguish when it dawned on engineers the cruel trick Apple and the standards committee had played. Mac OS X, being a Unix variant, had implemented wchar_t as UTF-32! All the cross platform code, code that used to work on Windows and Mac, no longer worked. Apple felt their pain, and issued this technical note, which essentially says: “instead of using wchar_t, which used to be cross platform before we destroyed it, use CFStringRef, which is not cross platform, has never been, and never will be. P.S. This is really your own fault for ever using wchar_t. Suckers.”

At the time that this was happening, I happened to work for Macromedia (now Adobe). Being the most important company that implements Flash, some of the Apple execs came down and talked to the Mac engineers at Macromedia. When the appropriate time came, I sprang into action demanding to know what would be done about wchar_t. There was stunned silence. “What’s wchar_t?” was the first answer. After explaining it, the next answer was “We don’t implement that.” After pointing them to their own documentation, the next answer was “Oh. Huh. Well, why did you use it? We don’t use that crap. Use CFString instead!” After slamming my head against the table, I attempted to explain wchar_t was used everywhere in our codebase, and CFString wasn’t cross platform. “Sure it is! It works on both Mac OS 9 and Mac OS X!”

The solution in the end for those duped into using wchar_t, is to go back and use unsigned short instead. Unfortunately, that means doing a lot find and replace (find: wchar_t replace: char16_t, where char16_t is typedef’d to unsigned short) and then re-implementing the wchar_t library (including wstring) for the new type. Yep. Reimplement the wchar_t library. The lucky jumped into a pit of rabid ice weasels, where they were torn from limb to limb. The unlucky had to repurpose all the old CodeWarrior MSL code to re-implement the wchar_t library as char16_t library.

The moral of the story is: don’t trust the standards committee. Especially on standards that aren’t really defined or when they start snickering behind your back. Usually that means they stuck a note on your back that says “Standardize me.” I’m not sure why that’s funny, but they think its hilarious. If you need to use a Unicode encoding, use UTF-8. You can just use char and std::string for that.

Besides, who doesn’t speak English?

I’m a Mac guy, but I’ve spent a large part of my time working for large software companies. That means that I usually work on a cross platform product that supports both Mac and Windows. The upshot of that is each product has to have a platform layer of some kind so that the correct Windows API’s are called on Windows and the correct Mac API’s are call on the Mac. In my tenure I’ve encountered several different ways of tackling this problem. Some worked very well, while others made me want to quit and become and mime.

The interesting thing to note here is that historically the Mac and Windows API’s weren’t all that different. Sure the Mac Toolbox calls started out in Pascal, but most apps use the C bindings these days. Both the Toolbox and Win32 have very similar ideas about windows, menus, and events, especially when comparing the Win32 API with the Carbon Toolbox. There’s definitely not a one to one mapping for everything, but it wasn’t all that hard to see where you could write a compatibility layer and support both.

There are a few different techniques/architectures for building cross platform frameworks. For this discussion I’ll ignore entire platforms like Java and Adobe Flash which employ virtual machines. I’ll limit discussion to natively compiled applications.

At the end of each section, I’ll rate the architecture. They’ll be evaluated based the cost of maintenance, the initial cost of development, and the user experience they advocate.

1. Emulation

   This is where you attempt to make one platform look like the other. For example you implement all the Mac Toolbox API’s that you use on Windows using the Win32 API. That way, you can program to the Mac API’s and it will just “magically” work on Windows. However, given platform popularity this usually means implementing MFC or Win32 on the Mac. Note that this isn’t emulation like what emulators do, but simply implementing a platform API on a different platform.

   This technique is unfortunately popular. There are some assumptions that go along with this:

   • The only thing required to develop for a platform is writing to the appropriate API.
   • If you emulate MFC on the Mac you can hire cheap MFC programmers and get a Mac version for free. (I’ve had managers tell me this with a straight face.)
   • It is cheaper and easier to maintain an emulation layer for a platform than the other techniques.
   • The platform on which the emulation is taking place is the only platform that suffers.
   • There is no other way to port an existing Windows application to another platform.

   It turns out every one of these assumptions are wrong.

   The first assumption assumes that the emulation layer is going to take care of everything and a Windows programmer will never have to know anything about the Mac. Unfortunately I can guarantee the emulation layer won’t be able to take care of everything and the Windows programmers will have to know something about the Mac. They will have to at least be able to build it on the Mac and figure out if their feature actually works there. But more than likely they’ll also have to know some basic UI guidelines and a general idea of how Mac users expect things to work. This includes the correct menu layout, correct dialog control layout, and the correct modifier keys to use.

   The second assumption implies that you can get away with only hiring Windows programmers and they will never ever have to touch the Mac code. Ha! The Mac emulation layer will have to be upgraded and maintained throughout the entire lifecycle of the product, almost daily. The emulation layer definitely is not a write-once-use-many-times sort of thing. You’ll start using new parts of the Win32 API that need to be implemented on the Mac, find bugs in the current implementation, and need to take advantage of new technology on the Mac.

   The conclusion to draw from the first two assumptions is any engineer will have to know or learn some basics about the Mac. They will at least need to be able to read some basic Mac GUI code and debug it.

   Thirdly, many people look at Emulation compared to, say, Core/Edge and draw the conclusion that emulation is half as cheap because the framework only exists on the Mac (in Core/Edge the framework exists on both platforms). They completely forget about the size of the framework. In Code/Edge you typically have a thin framework on each platform, where in Emulation you have a heavy framework on the Mac. Sure you don’t have anything on Windows, but you spending a lot more time on the Mac just maintaining the framework there.

   The fourth assumption people just seem to forget about. I mean, if you’re emulating Windows on the Mac, only the Mac port is going to suffer, right? Nope. Your cheap Windows engineer is going to start writing some code using some newfangled Win32 API and realize that its not implemented on the Mac emulation layer. Oops. They’d implement it on the Mac themselves, but you only hired cheap Windows programmers, remember? At this point, the engineer has to decide if they will ditch the new Win32 API’s or write a Windows only feature. So the Windows app suffers too because it becomes difficult to use the new Windows API’s. In other words, you’re really hurting both the Mac and Windows ports and the users are going to notice that.

   The last assumption is that there’s no other way to support another platform. If you’ve got a mature, existing Windows program its not going to be pleasant to port it no matter what option you choose. However, as I outline next, Emulation has a lot of hidden costs that end up making it much more expensive than the other options.

   Emulation is also non-incremental. You have to get a substantial amount of the framework written before you can start using it. This means there’s a large up front cost. In my experience it takes about ten or more engineers for about year to implement something reasonably resembling MFC on the Mac. It varies depending on how much Mac expertise you have. There’s also a large cost of upgrading this framework because you can’t upgrade one window at a time to use HIViews (or whatever), they have to be all upgraded at once.

   The cost of adding a platform (e.g. Linux) is the same as the initial development. That’s because the platform being emulated (e.g. Windows) isn’t abstracted out in any way.

   Finally, the Emulation framework is never complete. No matter how hard you work you’ll never get the Mac to look and behave like Windows even at the API level. It simply can’t be done. There are controls that the Mac has, that Windows doesn’t, and vice-versa. There are highlight and system colors that don’t have counterparts. This will trip up your Windows engineers who expect the Mac to behave like Windows. Since they don’t have any experience on the Mac they don’t know what to do, and you’re stuck.

   This is by far the worst way to approach cross platform development, and you should avoid it at all costs. I’ve worked on some large scale projects that used this approach and it was horrendous. The Windows programmers are encouraged to be clueless about the Mac and to hold up that development. It became difficult to do anything with the emulation layer except tread water. This lead to frustration and burn out. In other words, this approach is a great way to lose all your Mac engineers.

   Maintenance: D
   Initial Cost: F
   User experience: C

2. Bridge Pattern

   This is what Qt does. You write an entire application framework that entirely abstracts out all platform details, and provide a completely consistent cross platform interface. As a result, this architecture tends to like to do everything manually. Qt does not use native controls, but draws the controls itself. It is yet another API set to learn but its really the only one you need to know (almost). You write to this one API and it works on all the platforms.

   The benefit is obvious — cross platform consistency. You can just write to the framework and it will just magically work on every platform. Since the framework does everything manually, there is a lot of control to force the consistency.

   The are a few disadvantages to this approach. First and foremost, its usually expensive. Either you have to buy a framework that does this (like Qt) or develop it yourself. Unfortunately this approach doesn’t really lend itself to an incremental implementation. To do the most basic thing you need a fairly complete application framework on each of the platforms.

   Second, the framework might not be as native as you’d like on each of the platforms. Qt has this problem. To ensure the messaging and event system works completely the same on each platform, the framework will often reimplement large parts of the native platform. This makes it easier to program against, but hurts the user because it doesn’t feel completely native.

   Lastly, the framework is yet another API to learn. Both your Mac programmers and Windows programmers have to learn it, if they do not already know it. If, for whatever reason, you need to add a custom control, then you have to start digging around in platform specific code. In that case, the bridge abstraction does not help you. Therefore, having someone fluent in Qt but not Mac and/or Windows (or vice versa) isn’t useful.

   Maintenance: B
   Initial Cost: F
   User experience: C

3. Core/Edge

   This is a fairly simple idea. You write all your application logic in Standard C++ (or whatever common language and library). This is the “core” part, and should make up the majority of the code. In a MVC architecture, it should optimally be the model and controller parts. You also write core interfaces (class and function declarations) to access the platform specific code, such as the UI (aka the view part). The platform specific code, called the edge code, is written using a native framework or API’s.

   This strategy has several advantages. First, it gives the user the native feel and behavior of the platform because you end up using the native framework or API. Core/Edge doesn’t favor one platform over the other, but treats each platform equally. That means each can use the appropriate technologies with out have to emulate the other platform. It also means if one platform has a technology that another doesn’t its easy to use it without hurting the other platform.

   Second, it can be implemented incrementally. Since you’re using the native API’s you just start building the app on each platform as you normally would. As you start getting into common logic, or model classes, or controller classes, you simply switch to Core code. That means you write in your standard language with its standard library. (By standard I mean the language that is implemented on both platforms, like C++.) Typically your files will be broken in Core files and Edge files. If Core code needs to call back into Edge code, it calls into a Core interface. The Core interface only takes types defined by the standard language or the Core itself. The Core interface is then implemented by Edge code.

   The disadvantage with this approach is that it is easy to degenerate into the Edge/Edge approach. Doing this means you end up duplicating a lot of controller logic on each platform, or perhaps even view code. Some care has to be taken in ensuring the maximum amount of code is written such that it is Core.

   Another disadvantage is GUI resources. Since a native framework is used, then each platform is going to have a duplicate set of resources. For a given dialog, it will have to be re-layed out for each platform. If a control needs to be added, it will have be done once for each platform. This actually also provides an upside, which is each platform can use the native layout.

   In my experience, Core/Edge is the best architecture to use. It has a relatively low cost, high code reuse, and a very native feel.

   Maintenance: B
   Initial Cost: A
   User experience: A

4. Edge/Edge

   This is just what it sounds like. Usually there is some common application logic that is shared (as in Core/Edge) but the majority of UI controller code is just duplicated on each platform. There typically is not much effort to make sure any code compiles on both platforms, but each platform is built separately from fairly different codebases.

   The one advantage of this approach is that the application will feel very native, which the user will appreciate. The programmers for one platform can concentrate on making it highly integrated into that OS, and make use of new technologies.

   The major downside is very little code is leveraged or reused. That means most code ends up being duplicated, once for each platform. That can mean the application gets prohibitively expensive to maintain, and features cost a lot more to implement.

   Maintenance: D
   Initial Cost: B
   User experience: A

This list is not meant to be comprehensive, but to outline the architectures I have experienced. Applications can also use hybrid approaches that mix and match the architectures.

The future of cross platform architectures is going to be interesting. Recently the Mac and Windows API’s have been seriously diverging. Apple is pushing Cocoa, its Objective-C application framework. Not only is it in a different language, its much higher level than the Carbon API’s and has a different object model. Meanwhile Microsoft is pushing C# and .Net. Once again, a different language and a different object model. It is no longer as obvious as how to abstract out both API’s.

As a result, the Emulation architecture, and to a lesser extent the Bridge architecture, become a lot harder to implement. Cocoa is a much higher level than MFC, so implementing MFC in terms of it would be near impossible. The Bridge architecture would also have to find some high level abstractions to properly encapsulate both .Net and Cocoa. In reality, it would probably have to continue what it is doing now: using very low level API’s and do everything manually. Conversely, both the Core/Edge and Edge/Edge would be able to support both Cocoa and .Net. Since the framework is application specific and incremental, its abstractions can be changed to encapsulate the new native frameworks with relative ease.

I will freely admit I am very biased when it comes to cross platform GUI architectures. But that bias comes from me being forced to use each of these in large software systems. From the perspective of this Mac engineer, the Core/Edge architecture is by far the best.

It has recently come to my attention that there are currently no Apple based soap operas. Sure, there’s As the Apple Turns, but it hasn’t been updated in many months. I also have my doubts as to it being a real soap opera; nobody ever got amnesia. What kind of soap opera is that? There’s also Crazy Apple Rumors, which occasionally reads like a soap opera, but once again, no amnesia.

In order to fill this desperate need, and some free time, I am prepared to take up this gauntlet. We all have to make sacrifices sometime.

The scene opens with a product meeting about the new iPhone. Steve Jobs, Tony Fadell, and Tim Cook are all in attendance.

Jobs: Tony, how are we doing on the iPhone with a 52″ screen and telepathic interface?

Tony: Steve…

Jobs: Yes?

Tony: Its no use hiding behind your technical jargon.

Jobs: What?

Tony: Steve… don’t you know I love you?

Jobs: No! Its not possible!

Tony: Yes, it true!

Jobs: No, it can’t be. For you see… I’m not really Steve…

Tony: *gasp*

Jobs: I’m Steve’s evil twin quintuplet, Stewart!

Tony: You don’t mean… *gasp*

Jobs: Yes… *sob* Yes. I have amnesia!

Tony: *gasp*

Gil Amelio: *gasp*

Jobs: Who let you in?

Gil Amelio: Sorry. Here’s your pizza.

Jobs: Anyway… Tim, you’re looking a little large lately.

Tim: Yes… I’m… I’m pregnant.

Tony: *gasp*

Gil Amelio: *gasp*

Jobs: Are you done yet?

Gil Amelio: Um, yes. Sorry. I’m leaving.

Tim: Yes, Steve. I’m… I’m pregnant.

Tony: But… how??

Tim: Well, you see Tony, when a man loves a woman, they…

Tony: No, I mean, how does a man get pregnant? Don’t you need a uterus or something?

Tim: Oh. I’m not sure. But it happened to Arnold Schwarzenegger one time, so I’m sure its possible.

Jobs: Who’s the… father? …mother? …whatever… you know.

Tim: I don’t know. For you see… I have amnesia!

Tony: *gasp*

Jobs: Do you have to keep doing that?

Tony: Um… yes, actually. You put that in my contract. See? “Must gasp at any and all amnesia revelations.” Seemed a little weird at the time, but makes perfect sense now.

Jobs: Oh, right. I forgot… for you see.. I have amnesia!

Tony: *gasp*

Tim: Yes, yes, we covered that already. Shouldn’t we get back to the iPhone thingie?

Jobs: *sigh* Sure why not. What’s the problem again?

Tony: Um, we don’t know how to make one.

Jobs: Oh. Hmm.. that’s sort of a buzzkill. Does anyone else have any shocking revelations instead?

Jobs Evil Twin #4: Yes… I do.

Tony: *gasp* I thought you were dead!

Jobs Evil Twin #4: I was… in my heart. For you see… I have always loved you, Tim.

Tim: *sobs* I… I know… but our love is forbidden.

Jobs Evil Twin #4: Yes. That’s why I went into exile… to protect your feelings.

Jobs: Sounds boring.

Jobs Evil Twin #4: Um, yes. That’s why I made this.

Evil Twin #4 produces an iPhone.

Jobs: Sweet. I guess we’re done here.

Evil Twin #4: But wait… doesn’t anyone care that I have amnesia??

Tony: Not really.

Tune in next time, when someone else forgets they have an evil twin brother!

I probably need professional help.