version page 4 - Dave's Blog

The 'Is It UTF-8?' Quick and Dirty Test

I've found while debugging networking in IE its often useful to quickly tell if a string is encoded in UTF-8. You can check for the Byte Order Mark (EF BB BF in UTF-8) but, I rarely see the BOM on UTF-8 strings. Instead I apply a quick and dirty UTF-8 test that takes advantage of the well-formed UTF-8 restrictions.

Unlike other multibyte character encoding forms (see Windows supported character sets or IANA's list of character sets), for example Big5, where sticking together any two bytes is more likely than not to give a valid byte sequence, UTF-8 is more restrictive. And unlike other multibyte character encodings, UTF-8 bytes may be taken out of context and one can still know that its a single byte character, the starting byte of a three byte sequence, etc.

The full rules for well-formed UTF-8 are a little too complicated for me to commit to memory. Instead I've got my own simpler (this is the quick part) set of rules that will be mostly correct (this is the dirty part). For as many bytes in the string as you care to examine, check the most significant digit of the byte:

F:

This is byte 1 of a 4 byte encoded codepoint and must be followed by 3 trail bytes.

E:

This is byte 1 of a 3 byte encoded codepoint and must be followed by 2 trail bytes.

C..D:

This is byte 1 of a 2 byte encoded codepoint and must be followed by 1 trail byte.

8..B:

This is a trail byte.

0..7:

This is a single byte encoded codepoint.

Subst Allows Non-Letter Drive Letters

I knew that the command line tool subst would create virtual drives that map to existing directories but I didn't know that subst lets you name the virtual drives with characters that aren't US-ASCII letters. For instance you can run 'subst 4: C:\windows' and then 'more 4:\win.ini' to dump C:\windows\win.ini. This also works for non-US-ASCII characters like, "Ｃ" (aka U+FF23, Fullwidth Latin Capital Letter C), which when displayed by cmd.exe via some best fit style character conversions looks just like the regular US-ASCII 'C'. None of Explorer, IE, or the common file dialogs allow the use of these odd virtual drives -- just cmd.exe, so I'm not sure how this would ever be useful but I thought it was odd and I wanted to share.

Word Wrapping IE's Plain Text

If you view a plain text document in Internet Explorer 8, for instance the plain text version of Cory Doctorow's book Little Brother and press F12 to bring up the developer toolbar, you can see that IE simply takes the plain text, sticks it inside a

 tag, and renders it.  This means that word wrapping isn't supplied and the only line breaks that appear are those in the document.  However, since the text document is converted to HTML it means I can implement word wrap myself using a bookmarklet:
javascript:function ww() { var preTag = document.getElementsByTagName('pre')[0]; preTag.style.fontFamily="arial"; preTag.style.wordWrap='break-word'; }; ww();
After adding a favorite and setting the favorite's URL to the previous, I can view plain text documents, and select my Word Wrap favorite to apply word wrap and non-fixed width font.

Weekend Dinners: Old friends, Old library

Last Thursday I saw a bunch of college friends that I hadn't seen in a while, despite all of us working at Microsoft, and Saul and Ciera who were visiting. We had dinner at Typhoon! which I haven't been to in quite a while. Daniil and Val brought their cute child. I got to see Charlie and Matt who I'm not sure I've seen since my 25th birthday. There was much nerdiness. I need to remember to organize such a night myself sometime in near future so I don't have to wait another year to see them.

On the weekend Sarah and I went out to dinner at Carnegie's, a former public library in Ballard, Seattle that's now a restaurant. I saw the restaurant's website in Matt's delicious links and thought it looked interesting. The exterior and entryway look like a public library, but just inside its redone as a sort of modern version of french classical with a bar and two dining rooms. No pictures since my replacement camera only arrived today, but there are photos available. They serve french cuisine which was good and not as expensive as I would have expected. An interesting place, although its a bit of a drive and I'm not sure if we'll be going back soon.