ant page 16 - Dave's Blog

The 'Is It UTF-8?' Quick and Dirty Test

I've found while debugging networking in IE its often useful to quickly tell if a string is encoded in UTF-8. You can check for the Byte Order Mark (EF BB BF in UTF-8) but, I rarely see the BOM on UTF-8 strings. Instead I apply a quick and dirty UTF-8 test that takes advantage of the well-formed UTF-8 restrictions.

Unlike other multibyte character encoding forms (see Windows supported character sets or IANA's list of character sets), for example Big5, where sticking together any two bytes is more likely than not to give a valid byte sequence, UTF-8 is more restrictive. And unlike other multibyte character encodings, UTF-8 bytes may be taken out of context and one can still know that its a single byte character, the starting byte of a three byte sequence, etc.

The full rules for well-formed UTF-8 are a little too complicated for me to commit to memory. Instead I've got my own simpler (this is the quick part) set of rules that will be mostly correct (this is the dirty part). For as many bytes in the string as you care to examine, check the most significant digit of the byte:

F:

This is byte 1 of a 4 byte encoded codepoint and must be followed by 3 trail bytes.

E:

This is byte 1 of a 3 byte encoded codepoint and must be followed by 2 trail bytes.

C..D:

This is byte 1 of a 2 byte encoded codepoint and must be followed by 1 trail byte.

8..B:

This is a trail byte.

0..7:

This is a single byte encoded codepoint.

Subst Allows Non-Letter Drive Letters

I knew that the command line tool subst would create virtual drives that map to existing directories but I didn't know that subst lets you name the virtual drives with characters that aren't US-ASCII letters. For instance you can run 'subst 4: C:\windows' and then 'more 4:\win.ini' to dump C:\windows\win.ini. This also works for non-US-ASCII characters like, "Ｃ" (aka U+FF23, Fullwidth Latin Capital Letter C), which when displayed by cmd.exe via some best fit style character conversions looks just like the regular US-ASCII 'C'. None of Explorer, IE, or the common file dialogs allow the use of these odd virtual drives -- just cmd.exe, so I'm not sure how this would ever be useful but I thought it was odd and I wanted to share.

Back From Vegas

Sarah and I met up with Jon, Scott, Jesse, and Grib in Las Vegas last weekend and we had a fun time.

• I got to play the Monorail Song via YouTube on my phone while on the Las Vegas Monorail rather than just chanting monorail like last year.
• I didn't lose more gambling than I spent on food for the trip.
• Contrary to what some suggested, Sarah and I did not get married in Vegas.
• I finally saw a live Penn & Teller show and it was great!

Phone Replacement For Grocery Card

Another use for my new phone is as a replacement for my grocery card, those little plastic cards with a bar code on them that the grocery store gives you to track your purchasing habits. I've previously gone to great lengths to increase space in my pockets by removing infrequently used keys and reducing my wallet to the essentials. So I was glad to get rid of the QFC card and replace it with a photo of its bar code on my phone. Since the important part of the QFC card is the bar code which is just an image of black lines, if your phone has a camera and a screen with a reasonable resolution you can take a photo of the bar code and later display it to a reader. I've so far been able to try it once and successfully at a normal checkout line, but the reaction from the checkout lady was enough that I may in the future just keep a card in my car. She was very excited, asked me what kind of phone I had, called over another checkout person and generally made a large fuss. Also the checkout people generally don't mind giving me a new card if I don't have one with me.