solution page 2 - Dave's Blog

Blog Layout and Implementation Improvements

I've redone my blog's layout to remind myself how terrible CSS is -- err I mean to play with the more advanced features of CSS 2.1 which are all now available in IE8. As part of the new layout I've included my Delicious links by default but at a smaller size and I've replaced the navigation list options with Technical, Personal and Everything as I've heard from folks that that would actually be useful. Besides the layout I've also updated the back-end, switching from my handmade PHP+XSLT+RSS/Atom monster to a slightly less horrible PHP+DB solution. As a result everything should be much much faster including search which, incidentally, is so much easier to implement outside of XSLT.

PowerShell Scanning Script

I've hooked up the printer/scanner to the Media Center PC since I leave that on all the time anyway so we can have a networked printer. I wanted to hook up the scanner in a somewhat similar fashion but I didn't want to install HP's software (other than the drivers of course). So I've written my own script for scanning in PowerShell that does the following:

1. Scans using the Windows Image Acquisition APIs via COM
2. Runs OCR on the image using Microsoft Office Document Imaging via COM (which may already be on your PC if you have Office installed)
3. Converts the image to JPEG using .NET Image APIs
4. Stores the OCR text into the EXIF comment field using .NET Image APIs (which means Windows Search can index the image by the text in the image)
5. Moves the image to the public share

Here's the actual code from my scan.ps1 file:

param([Switch] $ShowProgress, [switch] $OpenCompletedResult)

$filePathTemplate = "C:\users\public\pictures\scanned\scan {0} {1}.{2}";
$time = get-date -uformat "%Y-%m-%d";

[void]([reflection.assembly]::loadfile( "C:\Windows\Microsoft.NET\Framework\v2.0.50727\System.Drawing.dll"))

$deviceManager = new-object -ComObject WIA.DeviceManager
$device = $deviceManager.DeviceInfos.Item(1).Connect();

foreach ($item in $device.Items) {
        $fileIdx = 0;
        while (test-path ($filePathTemplate -f $time,$fileIdx,"*")) {
                [void](++$fileIdx);
        }

        if ($ShowProgress) { "Scanning..." }

        $image = $item.Transfer();
        $fileName = ($filePathTemplate -f $time,$fileIdx,$image.FileExtension);
        $image.SaveFile($fileName);
        clear-variable image

        if ($ShowProgress) { "Running OCR..." }

        $modiDocument = new-object -comobject modi.document;
        $modiDocument.Create($fileName);
        $modiDocument.OCR();
        if ($modiDocument.Images.Count -gt 0) {
                $ocrText = $modiDocument.Images.Item(0).Layout.Text.ToString().Trim();
                $modiDocument.Close();
                clear-variable modiDocument

                if (!($ocrText.Equals(""))) {
                        $fileAsImage = New-Object -TypeName system.drawing.bitmap -ArgumentList $fileName
                        if (!($fileName.EndsWith(".jpg") -or $fileName.EndsWith(".jpeg"))) {
                                if ($ShowProgress) { "Converting to JPEG..." }

                                $newFileName = ($filePathTemplate -f $time,$fileIdx,"jpg");
                                $fileAsImage.Save($newFileName, [System.Drawing.Imaging.ImageFormat]::Jpeg);
                                $fileAsImage.Dispose();
                                del $fileName;

                                $fileAsImage = New-Object -TypeName system.drawing.bitmap -ArgumentList $newFileName 
                                $fileName = $newFileName
                        }

                        if ($ShowProgress) { "Saving OCR Text..." }

                        $property = $fileAsImage.PropertyItems[0];
                        $property.Id = 40092;
                        $property.Type = 1;
                        $property.Value = [system.text.encoding]::Unicode.GetBytes($ocrText);
                        $property.Len = $property.Value.Count;
                        $fileAsImage.SetPropertyItem($property);
                        $fileAsImage.Save(($fileName + ".new"));
                        $fileAsImage.Dispose();
                        del $fileName;
                        ren ($fileName + ".new") $fileName
                }
        }
        else {
                $modiDocument.Close();
                clear-variable modiDocument
        }

        if ($ShowProgress) { "Done." }

        if ($OpenCompletedResult) {
                . $fileName;
        }
        else {
                $result = dir $fileName;
                $result | add-member -membertype noteproperty -name OCRText -value $ocrText
                $result
        }
}

I ran into a few issues:

• MODI doesn't seem to be in the Office 2010 Technical Preview I installed first. Installing Office 2007 fixed that.
• The MODI.Document class, at least via PowerShell, can't be instantiated in a 64bit environment. To run the script on my 64bit OS I had to start powershell from the 32bit cmd.exe (C:\windows\syswow64\cmd.exe).
• I was planning to hook up my script to the scanner's 'Scan' button, but HP didn't get the button working for their Vista driver. Their workaround is "don't do that!".
• You must call Image.Dispose() to get .NET to release its reference to the corresponding image file.
• In trying to figure out how to store the text in the files comment, I ran into a dead-end trying to find the corresponding setter for GetDetailsOf which folks like James O'Neil use in PowerShell for interesting ends.

Thoughts on registerProtocolHandler in HTML 5

I'm a big fan of the concept of registerProtocolHandler in HTML 5 and in FireFox 3, but not quite the implementation. From a high level, it allows web apps to register themselves as handlers of an URL scheme so for (the canonical) example, GMail can register for the mailto URL scheme. I like the concept:

• Better integration of web apps with your system.
• Its easy for web apps to do.
• Links to URNs can now take the user to the sites the user prefers for the sort of thing identified by the URN. For example, if I have a physical address in HTML, instead of making that an http link to Yahoo Maps, I can make the link a geo scheme URI and those who follow the link will get their preferred mapping site that has registered for that scheme. Actually, looking at the geo scheme's RFC, maybe I'd rather use some other URN scheme to represent the physical location, but you get the point.

• There's no way to know if you are the handler for a particular URL scheme which is an important question for web app URL protocol handler authors.
• There's no way to fallback to an http URL in the case that a particular URL scheme isn't registered. A suggested solution to testing the registration of a scheme is for browsers to provide an additional script method to check if a scheme is registered. I don't like the idea of writing script that walks over all my page's links and rewrites them based on that method. I'd much rather see a declarative and backwards compatible fallback mechanism, although I don't know what that would look like.
• There's no way to register for a namespace within the urn scheme URI, the info scheme URI, or the tag scheme URI. I want to register info:lccn/... (Library of Congress Card Number identifiers) to LibraryThing or Amazon and I want to register urn:duri:... (dated URIs) to the Web Archive, among other things.


• Will this result in a proliferation of unregistered URL schemes with clashing namespaces? The ESW Wiki notes why this would be bad.
• And last, although this is nitpickier than the rest, I don't like the '%s' syntax used in the registration method. I'd much rather pass in an URL template, like the URL template used in OpenSearch. If an URL template is used for matching rather than registering against a particular URL scheme, this could also allow for registering a namespace within a URN. For example something along the lines of: registerProtocolHandler("info:lccn/{lccnID}", "htttp://www.librarything.com/search_works.php?q={lccnID}", "LibraryThing LCCN")

Phone Replacement For Grocery Card

Another use for my new phone is as a replacement for my grocery card, those little plastic cards with a bar code on them that the grocery store gives you to track your purchasing habits. I've previously gone to great lengths to increase space in my pockets by removing infrequently used keys and reducing my wallet to the essentials. So I was glad to get rid of the QFC card and replace it with a photo of its bar code on my phone. Since the important part of the QFC card is the bar code which is just an image of black lines, if your phone has a camera and a screen with a reasonable resolution you can take a photo of the bar code and later display it to a reader. I've so far been able to try it once and successfully at a normal checkout line, but the reaction from the checkout lady was enough that I may in the future just keep a card in my car. She was very excited, asked me what kind of phone I had, called over another checkout person and generally made a large fuss. Also the checkout people generally don't mind giving me a new card if I don't have one with me.

URI Fragment Info Roundup

Information about URI Fragments, the portion of URIs that follow the '#' at the end and that are used to navigate within a document, is scattered throughout various documents which I usually have to hunt down. Instead I'll link to them all here.

Definitions. Fragments are defined in the URI RFC which states that they're used to identify a secondary resource that is related to the primary resource identified by the URI as a subset of the primary, a view of the primary, or some other resource described by the primary. The interpretation of a fragment is based on the mime type of the primary resource. Tim Berners-Lee notes that determining fragment meaning from mime type is a problem because a single URI may contain a single fragment, however over HTTP a single URI can result in the same logical resource represented in different mime types. So there's one fragment but multiple mime types and so multiple interpretations of the one fragment. The URI RFC says that if an author has a single resource available in multiple mime types then the author must ensure that the various representations of a single resource must all resolve fragments to the same logical secondary resource. Depending on which mime types you're dealing with this is either not easy or not possible.

HTTP. In HTTP when URIs are used, the fragment is not included. The General Syntax section of the HTTP standard says it uses the definitions of 'URI-reference' (which includes the fragment), 'absoluteURI', and 'relativeURI' (which don't include the fragment) from the URI RFC. However, the 'URI-reference' term doesn't actually appear in the BNF for the protocol. Accordingly the headers like 'Request-URI', 'Content-Location', 'Location', and 'Referer' which include URIs are defined with 'absoluteURI' or 'relativeURI' and don't include the fragment. This is in keeping with the original fragment definition which says that the fragment is used as a view of the original resource and consequently only needed for resolution on the client. Additionally, the URI RFC explicitly notes that not including the fragment is a privacy feature such that page authors won't be able to stop clients from viewing whatever fragments the client chooses. This seems like an odd claim given that if the author wanted to selectively restrict access to portions of documents there are other options for them like breaking out the parts of a single resource to which the author wishes to restrict access into separate resources.

HTML. In HTML, the HTML mime type RFC defines HTML's fragment use which consists of fragments referring to elements with a corresponding 'id' attribute or one of a particular set of elements with a corresponding 'name' attribute. The HTML spec discusses fragment use additionally noting that the names and ids must be unique in the document and that they must consist of only US-ASCII characters. The ID and NAME attributes are further restricted in section 6 to only consist of alphanumerics, the hyphen, period, colon, and underscore. This is a subset of the characters allowed in the URI fragment so no encoding is discussed since technically its not needed. However, practically speaking, browsers like FireFox and Internet Explorer allow for names and ids containing characters outside of the defined set including characters that must be percent-encoded to appear in a URI fragment. The interpretation of percent-encoded characters in fragments for HTML documents is not consistent across browsers (or in some cases within the same browser) especially for the percent-encoded percent.

Text. Text/plain recently got a fragment definition that allows fragments to refer to particular lines or characters within a text document. The scheme no longer includes regular expressions, which disappointed me at first, but in retrospect is probably good idea for increasing the adoption of this fragment scheme and for avoiding the potential for ubiquitous DoS via regex. One of the authors also notes this on his blog. I look forward to the day when this scheme is widely implemented.

XML. XML has the XPointer framework to define its fragment structure as noted by the XML mime type definition. XPointer consists of a general scheme that contains subschemes that identify a subset of an XML document. Its too bad such a thing wasn't adopted for URI fragments in general to solve the problem of a single resource with multiple mime type representations. I wrote more about XPointer when I worked on hacking XPointer into IE.

SVG and MPEG. Through the Media Fragments Working Group I found a couple more fragment scheme definitions. SVG's fragment scheme is defined in the SVG documentation and looks similar to XML's. MPEG has one defined but I could only find it as an ISO document "Text of ISO/IEC FCD 21000-17 MPEG-12 FID" and not as an RFC which is a little disturbing.

AJAX. AJAX websites have used fragments as an escape hatch for two issues that I've seen. The first is getting a unique URL for versions of a page that are produced on the client by script. The fragment may be changed by script without forcing the page to reload. This goes outside the rules of the standards by using HTML fragments in a fashion not called out by the HTML spec. but it does seem to be inline with the spirit of the fragment in that it is a subview of the original resource and interpretted client side. The other hack-ier use of the fragment in AJAX is for cross domain communication. The basic idea is that different frames or windows may not communicate in normal fashions if they have different domains but they can view each other's URLs and accordingly can change their own fragments in order to send a message out to those who know where to look. IMO this is not inline with the spirit of the fragment but is rather a cool hack.

Personal Search with Yahoo Pipes

I've setup a minimal search page that uses a Yahoo Pipe to sort of search through my content. I say sort of search because I only get full text search over my recent item feeds and otherwise I just search over my tags.

To get real search I'm going to have to keep an archive of all my content on my own website. This is a pain but on the other hand it will let me easily backup my content or display old items on my page. Why didn't I just use a prebuilt solution?

IPv6 Roundup: Address Syntax on Windows

IPv6 address syntax consists of 8 groupings of colon delimited 16-bit hex values making up the 128-bit address. An optional double colon can replace any consecutive sequence of 0 valued hex values. For example the following is a valid IPv6 address: fe80::2c02:db79

Some IPv6 addresses aren't global and in those cases need a scope ID to describe their context. These get a '%' followed by the scope ID. For example the previous example with a scope ID of '8' would be: fe80::2c02:db79%8

IPv6 addresses in URIs may appear in the host section of a URI as long as they're enclosed by square brackets. For example: http://[fe80::2c02:db79]/. The RFC explicitly notes that there isn't a way to add a scope ID to the IPv6 address in a URI. However a draft document describes adding scope IDs to IPv6 addresses in URIs. The draft document uses the IPvFuture production from the URI RFC with a 'v1' to add a new hostname syntax and a '+' instead of a '%' for delimiting the scope id. For example: http://[v1.fe80::2c02:db79+8]/. However, this is still a draft document, not a final standard, and I don't know of any system that works this way.

In Windows XPSP2 the IPv6 stack is available but disabled by default. To enable the IPv6 stack, at a command prompt run 'netsh interface ipv6 install'. In Vista IPv6 is the on by default and cannot be turned off, while the IPv4 stack is optional and may be turned off by a command similar to the previous.

Once you have IPv6 on in your OS you can turn on IPv6 for IIS6 or just use IIS7. The address ::1 refers to the local machine.

In some places in Windows like UNC paths, IPv6 addresses aren't allowed. In those cases you can use a Vista DNS IPv6 hack that lives in the OS name resolution stack that transforms particularly crafted names into IPv6 addresses. Take your IPv6 address, replace the ':'s with '-'s and the '%' with an 's' and then append '.ipv6-literal.net' to the end. For example: fe80--2c02-db79s8.ipv6-literal.net. That name will resolve to the same example I've been using in Vista. This transformation occurs inside the system's local name resolution stack so no DNS servers are involved, although Microsoft does own the ipv6-literal.net domain name.

MSDN describes IPv6 addresses in URIs in Windows and I've described IPv6 addresses in URIs in IE7. File URIs in IE7 don't support IPv6 addresses. If you want to put a scope ID in a URI in IE7 you use a '%25' to delimit the scope ID and due to a bug you must have at least two digits in your scope ID. So, to take the previous example: http://[fe80::2c02:db79%2508]/. Note that its 08 rather than just 8.