client page 2 - Dave's Blog

WPAD Server Fiddler Extension

I've made a WPAD server Fiddler extension and in a fit of creativity I've named it: WPAD Server Fiddler Extension.

Of course you know about Fiddler, Eric's awesome HTTP debugger tool, the HTTP proxy that lets you inspect, visualize and modify the HTTP traffic that flows through it. And on the subject you've probably definitely heard of WPAD, the Web Proxy Auto Discovery protocol that allows web browsers like IE to use DHCP or DNS to automatically discover HTTP proxies on their network. While working on a particularly nasty WPAD bug towards the end of IE8 I really wished I had a way to see the WPAD requests and responses and modify PAC responses in Fiddler. Well the wishes of me of the past are now fulfilled by present day me as this Fiddler extension will respond to WPAD DHCP requests telling those clients (by default) that Fiddler is their proxy.

When I started working on this project I didn't really understand how DHCP worked especially with respect to WPAD. I won't bore you with my misconceptions: it works by having your one DHCP server on your network respond to regular DHCP requests as well as WPAD DHCP requests. And Windows I've found runs a DHCP client service (you can start/stop it via Start|Run|'services.msc', scroll to DHCP Client or via the command line with "net start/stop 'DHCP Client'") that caches DHCP server responses making it just slightly more difficult to test and debug my extension. If a Windows app uses the DHCP client APIs to ask for the WPAD option, this service will send out a DHCP request and take the first DHCP server response it gets. That means that if you're on a network with a DHCP server, my extension will be racing to respond to the client. If the DHCP server wins then the client ignores the WPAD response from my extension.

Various documents and tools I found useful while working on this:

• WPAD IETF draft document the never out of draft document describing WPAD
• DHCP RFC
• DHCP options RFC
• DHCP Wikipedia article which includes helpful byte layout tables for DHCP messages.
• Fiddler Wiki esp. the dev section but also the portion on the new preferences API.
• Fiddler Dev Docs and for those things not covered in the Fiddler docs, Visual Studio's name completion (or the .NET Reflector) usually served me well.
• Netmon a great network sniffing tool for all my non-HTTP network inspection needs.
• Tiny DHCP Server on CodeProject which implements a conventional DHCP server.

Comcast Digital Switch Impact on My Windows Media Center

Irritatingly out of line with what their commercials say, in my area Comcast, under the covers of the national broadcast digital switch, is sneaking in their own switch to digital, moving channels above 30 to their own digital format. Previously, I had Windows 7 Media Center running on a PC with a Hauppauge PVR500 which can decode two television signals at once setup to record shows I like. The XBox 360 works great as a Media Center client letting me easily watch the recorded shows over my home network on my normal TV.

Unfortunately with Comcast's change, now one needs a cable box or a Comcast digital to analog converter in order to view their signal, but Comcast is offering up to two free converters for those who'd like them. The second of my two free converters I hooked up to the Media Center PC and I got the IR Blaster that came with my Hauppauge out of the garage. I plugged in the USB IR Blaster to my PC, connected one of the IR transmitters to the 1st port on the IR Blaster, and sat the IR transmitter next to the converter's IR receiver. I went through the Media Center TV setup again and happily it was able to figure out how to correctly change the channel on the converter. So I can record now, however:

1. I can only record one thing at a time now
2. Changing the channel is slow taking many seconds (no flipping through channels for me)
3. The Hauppauge card can't know if the channel change worked. So if it tries to change to HBO (I get it for free with one of the Comcast packages) which is encrypted and the converted won't show, the channel doesn't change but the PC doesn't know it and ends up recording some other channel.

Flickr Visual Search in IE8

A while ago I promised to say how an xsltproc Meddler script would be useful and the general answer is its useful for hooking up a client application that wants data from the web in a particular XML format and the data is available on the web but in another XML format. The specific case for this post is a Flickr Search service that includes IE8 Visual Search Suggestions. IE8 wants the Visual Search Suggestions XML format and Flickr gives out search data in their Flickr web API XML format.

So I wrote an XSLT to convert from Flickr Search XML to Visual Suggestions XML and used my xsltproc Meddler script to actually apply this xslt.

After getting this all working I've placed the result in two places: (1) I've updated the xsltproc Meddler script to include this XSLT and an XML file to install it as a search provider - although you'll need to edit the XML to include your own Flickr API key. (2) I've created a service for this so you can just install the Flickr search provider if you're interested in having the functionality and don't care about the implementation. Additionally, to the search provider I've added accelerator preview support to show the Flickr slideshow which I think looks snazzy.

Doing a quick search for this it looks like there's at least one other such implementation, but mine has the distinction of being done through XSLT which I provide, updated XML namespaces to work with the released version of IE8, and I made it so you know its good.

Notes on Creating Internet Explorer Extensions in C++ and COM

Working on Internet Explorer extensions in C++ & COM, I had to relearn or rediscover how to do several totally basic and important things. To save myself and possibly others trouble in the future, here's some pertinent links and tips.

First you must choose your IE extensibility point. Here's a very short list of the few I've used:

• Explorer Bars and IE Toolbars
• IE Tools, IE Help, and IE Context Menu menu items
• IE Toolbar Buttons

Once you've created your COM object that implements IObjectWithSite and whatever other interfaces your extensibility point requires as described in the above links you'll see your SetSite method get called by IE. You might want to know how to get the top level browser object from the IUnknown site object passed in via that method.

After that you may also want to listen for some events from the browser. To do this you'll need to:

1. Implement the dispinterface that has the event you want. For instance DWebBrowserEvents2, or HTMLDocumentEvents, or HTMLWindowEvents2. You'll have to search around in that area of the documentation to find the event you're looking for.
2. Register for events using AtlAdvise. The object you need to subscribe to depends on the events you want. For example, DWebBrowserEvents2 come from the webbrowser object, HTMLDocumentEvents come from the document object assuming its an HTML document (I obtained via get_Document method on the webbrowser), and HTMLWindowEvents2 come from the window object (which oddly I obtained via calling the get_script method on the document object). Note that depending on when your SetSite method is called the document may not exist yet. For my extension I signed up for browser events immediately and then listened for events like NavigateComplete before signing up for document and window events.
3. Implement IDispatch. The Invoke method will get called with event notifications from the dispinterfaces you sign up for in AtlAdvise. Implementing Invoke manually is a slight pain as all the parameters come in as VARIANTs and are in reverse order. There's some ATL macros that may make this easier but I didn't bother.
4. Call AtlUnadvise at some point -- at the latest when SetSite is called again and your site object changes.

If you want to check if an IHTMLElement is not visible on screen due how the page is scrolled, try comparing the Body or Document Element's client height and width, which appears to be the dimensions of the visible document area, to the element's bounding client rect which appears to be its position relative to the upper left corner of the visible document area. I've found this to be working for me so far, but I'm not positive that frames, iframes, zooming, editable document areas, etc won't mess this up.

Be sure to use pointers you get from the IWebBrowser/IHTMLDocument/etc. only on the thread on which you obtained the pointer or correctly marshal the pointers to other threads to avoid weird crashes and hangs.

Obtaining the HTML document of a subframe is slightly more complicated then you might hope. On the other hand this might be resolved by the new to IE8 method IHTMLFrameElement3::get_contentDocument

Check out Eric's IE blog post on IE extensibility which has some great links on this topic as well.

Investigation of a Few Application Protocols (Updated)

Windows allows for application protocols in which, through the registry, you specify a URL scheme and a command line to have that URL passed to your application. Its an easy way to hook a webbrowser up to your application. Anyone can read the doc above and then walk through the registry and pick out the application protocols but just from that info you can't tell what the application expects these URLs to look like. I did a bit of research on some of the application protocols I've seen which is listed below. Good places to look for information on URI schemes: Wikipedia URI scheme, and ESW Wiki UriSchemes.

URI Fragment Info Roundup

Information about URI Fragments, the portion of URIs that follow the '#' at the end and that are used to navigate within a document, is scattered throughout various documents which I usually have to hunt down. Instead I'll link to them all here.

Definitions. Fragments are defined in the URI RFC which states that they're used to identify a secondary resource that is related to the primary resource identified by the URI as a subset of the primary, a view of the primary, or some other resource described by the primary. The interpretation of a fragment is based on the mime type of the primary resource. Tim Berners-Lee notes that determining fragment meaning from mime type is a problem because a single URI may contain a single fragment, however over HTTP a single URI can result in the same logical resource represented in different mime types. So there's one fragment but multiple mime types and so multiple interpretations of the one fragment. The URI RFC says that if an author has a single resource available in multiple mime types then the author must ensure that the various representations of a single resource must all resolve fragments to the same logical secondary resource. Depending on which mime types you're dealing with this is either not easy or not possible.

HTTP. In HTTP when URIs are used, the fragment is not included. The General Syntax section of the HTTP standard says it uses the definitions of 'URI-reference' (which includes the fragment), 'absoluteURI', and 'relativeURI' (which don't include the fragment) from the URI RFC. However, the 'URI-reference' term doesn't actually appear in the BNF for the protocol. Accordingly the headers like 'Request-URI', 'Content-Location', 'Location', and 'Referer' which include URIs are defined with 'absoluteURI' or 'relativeURI' and don't include the fragment. This is in keeping with the original fragment definition which says that the fragment is used as a view of the original resource and consequently only needed for resolution on the client. Additionally, the URI RFC explicitly notes that not including the fragment is a privacy feature such that page authors won't be able to stop clients from viewing whatever fragments the client chooses. This seems like an odd claim given that if the author wanted to selectively restrict access to portions of documents there are other options for them like breaking out the parts of a single resource to which the author wishes to restrict access into separate resources.

HTML. In HTML, the HTML mime type RFC defines HTML's fragment use which consists of fragments referring to elements with a corresponding 'id' attribute or one of a particular set of elements with a corresponding 'name' attribute. The HTML spec discusses fragment use additionally noting that the names and ids must be unique in the document and that they must consist of only US-ASCII characters. The ID and NAME attributes are further restricted in section 6 to only consist of alphanumerics, the hyphen, period, colon, and underscore. This is a subset of the characters allowed in the URI fragment so no encoding is discussed since technically its not needed. However, practically speaking, browsers like FireFox and Internet Explorer allow for names and ids containing characters outside of the defined set including characters that must be percent-encoded to appear in a URI fragment. The interpretation of percent-encoded characters in fragments for HTML documents is not consistent across browsers (or in some cases within the same browser) especially for the percent-encoded percent.

Text. Text/plain recently got a fragment definition that allows fragments to refer to particular lines or characters within a text document. The scheme no longer includes regular expressions, which disappointed me at first, but in retrospect is probably good idea for increasing the adoption of this fragment scheme and for avoiding the potential for ubiquitous DoS via regex. One of the authors also notes this on his blog. I look forward to the day when this scheme is widely implemented.

XML. XML has the XPointer framework to define its fragment structure as noted by the XML mime type definition. XPointer consists of a general scheme that contains subschemes that identify a subset of an XML document. Its too bad such a thing wasn't adopted for URI fragments in general to solve the problem of a single resource with multiple mime type representations. I wrote more about XPointer when I worked on hacking XPointer into IE.

SVG and MPEG. Through the Media Fragments Working Group I found a couple more fragment scheme definitions. SVG's fragment scheme is defined in the SVG documentation and looks similar to XML's. MPEG has one defined but I could only find it as an ISO document "Text of ISO/IEC FCD 21000-17 MPEG-12 FID" and not as an RFC which is a little disturbing.

AJAX. AJAX websites have used fragments as an escape hatch for two issues that I've seen. The first is getting a unique URL for versions of a page that are produced on the client by script. The fragment may be changed by script without forcing the page to reload. This goes outside the rules of the standards by using HTML fragments in a fashion not called out by the HTML spec. but it does seem to be inline with the spirit of the fragment in that it is a subview of the original resource and interpretted client side. The other hack-ier use of the fragment in AJAX is for cross domain communication. The basic idea is that different frames or windows may not communicate in normal fashions if they have different domains but they can view each other's URLs and accordingly can change their own fragments in order to send a message out to those who know where to look. IMO this is not inline with the spirit of the fragment but is rather a cool hack.