Inspired by a discussion on the relative merits of scala vs. java which brought up sorting a list of strings based on the result of a call to toLowercase(), I offer you a trip around the world to convince you that string comparison is a lot more complex.

If you are a java user and don’t want to read it all, the take-away lesson is simple: You should be using java.text.Collator which is better than Collections.sort(list, String.CASE_INSENSITIVE_ORDER) which is better than just calling toLowerCase() on everything.

To explain why string comparison is so much more nuanced, we’ll visit three interesting locales, each throwing a unique wrench in the works.

Example 1: The Eszett character

In german, a certain ligature for two consecutive s characters has been used so much for so long it’s considered its own unique character. The Eszett or Scharfes s: ß. It shows up all over the place in german, for example in the word street, which is straße in german. The Eszett is somewhat unique in that it has no capital form. Words don’t start with double s so there has never been a reason to capitalize it. Instead, if you ask a german to write street in all caps, they’d write STRASSE.

Uhoh. That’s not a reversible operation! The lowercase form of STRASSE is strasse, and even knowing that the locale is german, you can’t just guess that double-s ALWAYS becomes the ß character. You’d need a dictionary. Arguably you’re better off doing that conversion than not doing it if you know its german (more likely), but it won’t be a sure thing. Nevertheless, trying to check if straße and STRASSE are two equal strings under case insensitive comparison should return true, and not false, but while that’s in the cards, using .toLowerCase() as a proxy for case insensitive comparison is never going to get us that.

Java actually gets this right, sort of. trying to uppercase straße will in fact produce STRASSE even when just using .toUpperCase(), you don’t have to specify Locale.GERMANY. You can experiment by using \u00DF which is the (lowercase) Eszett. There is such a thing as a capital Eszett in unicode, which nobody in germany uses and is clearly a silly idea: You don’t hack a living, breathing language to get around i18n problems. If that’s feasible we might as well tell the world to stop whining and only use english when using machines.

Here’s the problem: String.CASE_INSENSITIVE_ORDER does not in fact return ’0′ (indicating equality) when comparing straße and STRASSE, even though every single language that uses the ß (german. That’s it!) says those two are case-insensitive equal. Nevertheless, that does not excuse the brokenness of the ‘toLowerCase() is an alternative for case insensitive comparison’ meme.

NB: My thanks go to dpark who spotted a problem in an earlier version of this post where I erroneously reported java did in fact use the invented uppercase variant.

Example 2: The turkish dotless/dotted i, and the somewhat famous PHP breaks on turkish computers problem

The ß vs. SS issue indicates that toLowerCase() isn’t a valid replacement for case insensitive comparison, but it gets worse. You can’t, in fact, say anything whatsoever about casing, or case insensitive equality, without knowing the locale you’re operating in. In turkish there’s not one ‘i’ but two: The dotted i and the dotless i. Where this gets crazy is the capital forms of those. The capital form of ‘i’ is a dotted capital I, and the capital form of dotless ‘i’ is our normal, familiar capital I, which doesn’t have a dot in it. This even messes with kerning (fi is a common kerning where the tip of the curvey top of the f is joined with the i’s dot. That’s not appropriate in turkish, where that dot is important). This means “i” and “I” are not equal in case-insensitive comparison in the Turkish locale. toLowerCase() comparison gets this wrong, of course. toLowerCase(new Locale("TR")) would actually get this right. String.CASE_INSENSITIVE_ORDER gets this wrong, because you can’t give it a locale. For natural language comparison (and why are you upper/ lowering strings in the first place, if not because youre doing natural language comparison?), tLC(), tUC(), String.CASE_INSENSITIVE_ORDER – are all inherently broken. Java gets this a little right and offers Locale-based variants of tLC() and tUC(), as I hinted at earlier.

That PHP we always like to bash on? One of the things PHP does (or perhaps did, I don’t keep up with it) wrong for the longest of times was that it would completely lose its marbles if the system’s locale was set to turkish. It would just fail to work. PHP identifiers are defined as case insensitive, and PHP implemented this by toLowerCase()ing everything, using the platform default encoding. This turns “FILE” into “fıle” which is not equal to “file”, and thus running most PHP code on a machine configured with turkish locale breaks, if the PHP is in english. This is a famous example of the universally lamented “it works for me” attitude. So, yes, making this mistake has dire consequences.

Example 3: Ascii hacks

To make matters worse still, lets think about why one would want to do a case insensitive comparison in the first place. Presumably because there’s some user input of some sort that needs to be compared, and you don’t want to bother the user with case sensitivity. However, if thats the aim, “case insensitive” is entirely the wrong idea. There are a bajillion systems around that can’t deal with unicode. For example, if many ‘name’ forms don’t even accept a dash in your last name, how many do you think accept a “ü” in it? And yet plenty of german folks are called “Müller”. The ‘fix’ these poor saps have used for years and years is to write the canonical ascii alternative for their funky character. ß becomes ss, ü because ue, etcetera. I’m betting that if you intend for “JOE” and “joe” to be equal, then you should consider “müller” and “mueller” equal too.

Whats really needed is a human inputted string comparator. Optimally speaking such a tool will first canonicalize each string, turning for example dotless i of any capitalization into a dotted lowercase i, turns ß into ss, ü into ue, etcetera, and only then compare the strings. I’m not even sure this can be done properly without knowing a locale, but you could do a lot better than String.CASE_INSENSITIVE_ORDER that way, and far better still versus comparing the .toLowerCase() versions of any two given strings. Java does in fact have something for this: java.text.Collator. This indeed does more or less what I just described, and it is in fact what one ought to be using. If I were you, I’d make a pmd plugin right now that checks for usage of locale-less tLC and tUC, as well as String.CASE_INSENSITIVE_ORDER, and flag them all as warnings.

Bonus Example: Lower case. Upper case. And Title case??

Astute folks may have observed that Character.toTitleCase() exists in the java standard library.

What’s that you ask? Well, in some languages, there are single characters that look like 2 characters. For example the ‘dz’. That’s one character in some languages. This is very similar to the germans who have enshrined the much used ß ligature into a unique character. However, unlike the ß which has pretty much lost all resemblance to the original character, and which can never appear at the beginning of a word, these characters are mostly just a kerned version of the original, and CAN appear at the beginning of the word. In unicode speak these are called digraphs. These have 3 and not 2 capitalization forms: all lowercase, all uppercase, and the first part of the digraph uppercased, the second part lowercased. This is what you’d use when you want a word with just the first letter capitalized, and you use the all-caps version only if you need an all-caps rendering of the word (i.e. rarely). This obscure little factoid actually was useful for me when writing lombok: The method that turns “foo” into “getFoo” will use toTitleCase() and not toUpperCase().

And there ends our trip round the world. I hope you enjoyed it. Though now you know: That feeling of despair when someone mentions i18n? You’re entirely correct in feeling it. It’s a pain in the tusch!

I recently got an email about a side-project, org.microformats.hCard, about the src zip being broken. Once I got that sorted out, an almost immediately followup that a 1.6 dependency snuck in there, including detailed reports.

Earlier, someone contributed XFN support for it. And, for lombok, we’ve recently received patches for netbeans and JDK7 support. The current lombok ‘hailbunny’ beta is out with netbeans support thanks to these patches (Jan Lahoda: You rock!). You can download the beta here.

Could be a coincidence, but this is the first time that the make it open source and people send you patches theory is starting to work out, and as a developer, that kind of feedback is just awesome.

So, if you are a user of an open source project and you find a bug or think of a nice feature, implement it, send a patch, and odds are you’ll get the eternal gratitude of the developer, and you’ll likely see your patch integrated into the main release.

Patches rock!

Every so often a well meaning famous person re-invents the notion of the environmentalist Pascal’s Wager. The case being made works like this:

The theory then goes: Even if the odds that the earth is actually doomed unless we make some drastic changes is very low – then the above table should make it obvious that anything above 0% is more than enough reason to get drastic and do get started right now. After all, multiplying the odds will always get you to -infinity if the world decides to keep going as we are.

This argument is often touted by crackpots. See Exhibit A. That’s okay – it takes all sorts to make a world. However, I get personally miffed when otherwise very smart people fall for this logical fallacy. I get even more miffed when they somehow proudly proclaim how they repurposed Pascal’s Wager. Pascal’s Wager is itself a fairly obvious fallacy. Normally, smart people are supposed to be able to add two and two together. Maybe that M. Night Shyamalan movie was on the something and tree hugging makes your brains fall out of your ears. I don’t know what it is.

Thus, a public service for all those who experience temporary bouts of idiocy when it comes to the environment. It happens to all of us, don’t feel bad.

I’ll fix that table for you, should make it obvious why you should stop talking about Pascal already:

If you think that particular doom scenario is preposterous, I will posit that earth being utterly doomed when we don’t fix up the environment is similarly preposterous. There are also lots and lots of other doom scenarios: A killer virus, aliens, global thermonuclear war, a world-wide revolt due to male overpopulation and religious fanaticism due to continued poverty, corruption, and civil war in the third world – pick your poison. You’ll find plenty of theories about how the earth could end if we aren’t careful.

So, we should fix all of that right?

Right. Except earth is a place of finite resource. Human creativity is still a finite resource. We can’t focus on all of that equally. We have to make choices.

And this is where I, and other sane and right thinking individuals like Bjorn Lomborg – who is one of the few true environmentalists I know of, stand: The only way to get out of this dilemma is to create win-win scenarios. Running around like a headless chicken screaming bloody murder about the environment, using cockamamie theories like the Environmentalist’s Pascal’s Wager, the more we’re going to really risk ending up doomed. There are so many things we could do that would actually save the environment, and be a net positive effect on mankind economically and scientifically, but virtually none of it gets done. Instead we spend all our time lobbying for people to become just as treehugger crazy and for utterly ineffective corn ethanol. D’oh.

There’s also a long-standing tradition in certain more fanatical environmentalist groups to be completely unreasonable, and make downright retarded decisions. Every hip american buying a Prius is one fine example: Considering the trade-in value is so worthless that most of them hit the scrapyards inside of 15 years (those batteries just don’t last all that long), you should probably just buy a Hummer. Mother Earth would be more grateful. Inane FUD spreading of nuclear power plants is another one: It’s a very tough choice, but France has been doing just fine with it so far. If the french can make nuclear safe enough, then what the heck is everybody whining about? We should at least take it a bit more seriously.

Apple is being torched almost daily for not being green enough. This is the company that make the notebook computer as the only computer you own popular. The amount of power and material saved is enormous. Green peace and company should grovel at Steve Jobs’ knees and thank him for the good he’s done. Of every soul I meet that proclaims they are an environmentalist, 99% of them are short sighted bags of hot air. It annoys me so.

The latest fine example of environmentalists being absolute idiots is 2 google searches generate 7 grams of co2.

Riiiight – and driving to the library and spending 5 hours looking it up there is much better for the environment. How short-sighted can you get? Google, and the internet in general, is the best most scalable thing that has ever happened to the environment.

I’m not a global warming denialist. I strongly feel that we, as mankind, ought to be doing more. I just get annoyed when otherwise smart people turn into blithering idiots, is all. Please, for the sake of this planet – use some common sense. I thank you.