modi page 2 - Dave's Blog

WPAD Server Fiddler Extension

I've made a WPAD server Fiddler extension and in a fit of creativity I've named it: WPAD Server Fiddler Extension.

Of course you know about Fiddler, Eric's awesome HTTP debugger tool, the HTTP proxy that lets you inspect, visualize and modify the HTTP traffic that flows through it. And on the subject you've probably definitely heard of WPAD, the Web Proxy Auto Discovery protocol that allows web browsers like IE to use DHCP or DNS to automatically discover HTTP proxies on their network. While working on a particularly nasty WPAD bug towards the end of IE8 I really wished I had a way to see the WPAD requests and responses and modify PAC responses in Fiddler. Well the wishes of me of the past are now fulfilled by present day me as this Fiddler extension will respond to WPAD DHCP requests telling those clients (by default) that Fiddler is their proxy.

When I started working on this project I didn't really understand how DHCP worked especially with respect to WPAD. I won't bore you with my misconceptions: it works by having your one DHCP server on your network respond to regular DHCP requests as well as WPAD DHCP requests. And Windows I've found runs a DHCP client service (you can start/stop it via Start|Run|'services.msc', scroll to DHCP Client or via the command line with "net start/stop 'DHCP Client'") that caches DHCP server responses making it just slightly more difficult to test and debug my extension. If a Windows app uses the DHCP client APIs to ask for the WPAD option, this service will send out a DHCP request and take the first DHCP server response it gets. That means that if you're on a network with a DHCP server, my extension will be racing to respond to the client. If the DHCP server wins then the client ignores the WPAD response from my extension.

Various documents and tools I found useful while working on this:

• WPAD IETF draft document the never out of draft document describing WPAD
• DHCP RFC
• DHCP options RFC
• DHCP Wikipedia article which includes helpful byte layout tables for DHCP messages.
• Fiddler Wiki esp. the dev section but also the portion on the new preferences API.
• Fiddler Dev Docs and for those things not covered in the Fiddler docs, Visual Studio's name completion (or the .NET Reflector) usually served me well.
• Netmon a great network sniffing tool for all my non-HTTP network inspection needs.
• Tiny DHCP Server on CodeProject which implements a conventional DHCP server.

Android eBook Reader And Makers

I was reading Makers, Cory Doctorow's latest novel, as it was serialized on Tor's website but with no ability to save my place within a page I set out to find a book reading app for my G1 Android phone. I stopped looking once I found Aldiko. Its got bookmarks within chapters, configurable fonts, you can look-up words in a dictionary, and has an easy method to download public domain and creative common books. I was able to take advantage of Aldiko's in-app book download system to get Makers onto my phone so I didn't have to bother with any conversion programs etc, and I didn't have to worry about spacing or layout, the book had the correct cover art, and chapter delimiters. I'm very happy with this app and finished reading Makers on it.

Makers is set in the near future and features teams of inventors, networked 3d printers, IP contention, body modifications, and Disney -- just the sort of thing you'd expect from a Cory Doctorow novel. The tale seems to be an allegory for the Internet including displacing existing businesses and the conflict between the existing big entertainment IP owners and the plethora of fans and minor content producers. The story is engaging and the characters filled out and believable. I recommend Makers and as always its Creative Commons so go take a look right now.

PowerShell Scanning Script

I've hooked up the printer/scanner to the Media Center PC since I leave that on all the time anyway so we can have a networked printer. I wanted to hook up the scanner in a somewhat similar fashion but I didn't want to install HP's software (other than the drivers of course). So I've written my own script for scanning in PowerShell that does the following:

1. Scans using the Windows Image Acquisition APIs via COM
2. Runs OCR on the image using Microsoft Office Document Imaging via COM (which may already be on your PC if you have Office installed)
3. Converts the image to JPEG using .NET Image APIs
4. Stores the OCR text into the EXIF comment field using .NET Image APIs (which means Windows Search can index the image by the text in the image)
5. Moves the image to the public share

Here's the actual code from my scan.ps1 file:

param([Switch] $ShowProgress, [switch] $OpenCompletedResult)

$filePathTemplate = "C:\users\public\pictures\scanned\scan {0} {1}.{2}";
$time = get-date -uformat "%Y-%m-%d";

[void]([reflection.assembly]::loadfile( "C:\Windows\Microsoft.NET\Framework\v2.0.50727\System.Drawing.dll"))

$deviceManager = new-object -ComObject WIA.DeviceManager
$device = $deviceManager.DeviceInfos.Item(1).Connect();

foreach ($item in $device.Items) {
        $fileIdx = 0;
        while (test-path ($filePathTemplate -f $time,$fileIdx,"*")) {
                [void](++$fileIdx);
        }

        if ($ShowProgress) { "Scanning..." }

        $image = $item.Transfer();
        $fileName = ($filePathTemplate -f $time,$fileIdx,$image.FileExtension);
        $image.SaveFile($fileName);
        clear-variable image

        if ($ShowProgress) { "Running OCR..." }

        $modiDocument = new-object -comobject modi.document;
        $modiDocument.Create($fileName);
        $modiDocument.OCR();
        if ($modiDocument.Images.Count -gt 0) {
                $ocrText = $modiDocument.Images.Item(0).Layout.Text.ToString().Trim();
                $modiDocument.Close();
                clear-variable modiDocument

                if (!($ocrText.Equals(""))) {
                        $fileAsImage = New-Object -TypeName system.drawing.bitmap -ArgumentList $fileName
                        if (!($fileName.EndsWith(".jpg") -or $fileName.EndsWith(".jpeg"))) {
                                if ($ShowProgress) { "Converting to JPEG..." }

                                $newFileName = ($filePathTemplate -f $time,$fileIdx,"jpg");
                                $fileAsImage.Save($newFileName, [System.Drawing.Imaging.ImageFormat]::Jpeg);
                                $fileAsImage.Dispose();
                                del $fileName;

                                $fileAsImage = New-Object -TypeName system.drawing.bitmap -ArgumentList $newFileName 
                                $fileName = $newFileName
                        }

                        if ($ShowProgress) { "Saving OCR Text..." }

                        $property = $fileAsImage.PropertyItems[0];
                        $property.Id = 40092;
                        $property.Type = 1;
                        $property.Value = [system.text.encoding]::Unicode.GetBytes($ocrText);
                        $property.Len = $property.Value.Count;
                        $fileAsImage.SetPropertyItem($property);
                        $fileAsImage.Save(($fileName + ".new"));
                        $fileAsImage.Dispose();
                        del $fileName;
                        ren ($fileName + ".new") $fileName
                }
        }
        else {
                $modiDocument.Close();
                clear-variable modiDocument
        }

        if ($ShowProgress) { "Done." }

        if ($OpenCompletedResult) {
                . $fileName;
        }
        else {
                $result = dir $fileName;
                $result | add-member -membertype noteproperty -name OCRText -value $ocrText
                $result
        }
}

I ran into a few issues:

• MODI doesn't seem to be in the Office 2010 Technical Preview I installed first. Installing Office 2007 fixed that.
• The MODI.Document class, at least via PowerShell, can't be instantiated in a 64bit environment. To run the script on my 64bit OS I had to start powershell from the 32bit cmd.exe (C:\windows\syswow64\cmd.exe).
• I was planning to hook up my script to the scanner's 'Scan' button, but HP didn't get the button working for their Vista driver. Their workaround is "don't do that!".
• You must call Image.Dispose() to get .NET to release its reference to the corresponding image file.
• In trying to figure out how to store the text in the files comment, I ran into a dead-end trying to find the corresponding setter for GetDetailsOf which folks like James O'Neil use in PowerShell for interesting ends.

OpenSearchDescriptionToHTML Tool

I've made an OpenSearchDescriptionToHTML XSLT that given an OpenSearch description file produces HTML that describes that file, lets you install it, or search with it. For example, here's a Google OpenSearch description that uses my OpenSearchDescriptionToHTML XSLT.

I had just created an OpenSearch description for WolframAlpha at work and was going about the process of adding another install link to my search provider page so that I could install it. Thinking about it, I realized I could apply an XSLT to the OpenSearch description XML to produce the HTML automatically so I wouldn't have to modify additional documents everytime I create and want to install a new OpenSearch description. While I was in there writing the XSLT I figure why not let the user try out searching with the OpenSearch description file too. And lastly I made the XSLT apply to itself to produce HTML describing its own usage.

Incidentally, I added WolframAlpha at work to replace my FileInfo search provider for the purposes of searching for information about particular Unicode characters. For instance, look at WolframAlpha's lovely output for this search for "Bopomofo zh".

Stop For Me, Its the Claw

sequelguy posted a photo:

Humorous modified sign: "Stop for me its the law" changed to "... its the claw" with a claw added to the stick figures arm.

URI Addressable Text Adventure Games

This post is about creating a server side z-code interpreter that represents game progress in the URI. Try it with the game Lost Pig.

I enjoy working on URIs and have the mug to prove it. Along those lines I've combined thoughts on URIs with interactive fiction. I have a limited amount of experience with Inform which generates Z-Code so I'll focus on pieces written in that. Of course we can already have URIs identifying the Z-Code files themselves, but I want URIs to identify my place in a piece of interactive fiction. The proper way to do this would be to give Z-Code its own mimetype and associate with that mimetype the format of a fragment that would contain the save state of user's interactive fiction session. A user would install a browser plugin that would generate URIs containing the appropriate fragment while you play the IF piece and be able to load URIs identifying Z-Code files and load the save state that appears in the fragment.

But all of that would be a lot of work, so I made a server side version that approximates this. On the Web Frotz Interpreter page, enter the URI of a Z-Code file to start a game. Enter your commands into the input text box at the bottom and you get a new URI after every command. For example, here's the beginning of Zork. I'm running a slightly modified version of the Unix version of Frotz. Baf's Guide to the IF Archive has lists of IF games to try out.

There are two issues with this thought, the first being the security issues with running arbitrary z-code and the second is the practical URI length limit of about 2K in IE. From the Z-Code standard and the Frotz source it looks like 'save' and 'restore' are the only commands that could do anything interesting outside of the Z-Code virtual machine. As for the length-limit on URIs I'm not sure that much can be done about that. I'm using a base64 encoded copy of the compressed input stream in the URI now. Switching to the actual save state might be smaller after enough user input.

Chumby will be cool, despite its name

Old Miscellaneous Thoughts

XSL Identity Transfom

Backup Notes

sudo fdisk /dev/sda

m

help

p

print current partitions

d

delete current partition

n

create new partition (I used the defaults)

w

write changes and exit

sudo mkfs.ext3 /dev/sda1

sudo vim /etc/fstab
# added line to end:
/dev/sda1 /media/backup ext3 rw,user,noauto 0 0

mount /media/backup
sudo mkdir /media/backup/users
sudo mkdir /media/backup/users/dave
sudo chown dave:dave /media/backup/users/dave

mount /media/backup
cp -Rv /home/dave/svn /media/backup/users/dave/
umount /media/backup

svn co file:///home/dave/svn/trunk/web/dave%40deletethis.net/public_html

svn help

help

svn list file:///home/dave/svn/

list all files in root dir of svn depot

svn list -R file:///home/dave/svn/

list all files in svn depot

svn list -R file:///home/dave/svn/ | grep \/$

list all directories

svn status

List status of all files in the working copy directory as in - modified, not in repository, etc

svn update

Brings the working copy up to date wrt the repository

svn commit

Commit changes from the working copy to the repository

svn add / move / delete

Perform the specified action -- occurs immediately

putty.exe dave@deletethis.net
cd .ssh
vim authorized_keys # leave the putty window open for now
puttygen.exe
Click the 'generate' button
Move the mouse around until finished
Copy text in 'Public key for pasting into OpenSSH authorized_keys file:' to putty window & save & close putty window
Enter Key passphrase & Comment in puttygen
Save the private key somewhere private
pageant.exe
'Add Key' the private key just saved.

svn+ssh://dave@deletethis.net/home/dave/svn/trunk/web/dave%40deletethis.net/public_html/

set SVN_SSH=C:\\users\\dave\\bin\\putty\\plink.exe