API page 4 - Dave's Blog

I've been working on the Glitch Helperator. It is a collection of tools and things I've put together for Glitch. It has a few features that I haven't seen elsewhere including:

Favorite Streets

A notebook in which you can save information about interesting streets and later use it to find your way back to them.

Birthday

Find out how old your Glitch is and the date of your next birthday in Glitch time or Earth time.

API Update History

A history of changes to the streets, skills and achievements of Glitch noting when new ones are added and when existing ones are changed.

It also has an interactive skill tree, find nearest feature tool, and achievement display. If you play Glitch, check it out.

One of the more limiting issues of writing client side script in the browser is the same origin limitations of XMLHttpRequest. The latest version of all browsers support a subset of CORS to allow servers to opt-in particular resources for cross-domain access. Since IE8 there's XDomainRequest and in all other browsers (including IE10) there's XHR L2's cross-origin request features. But the vast majority of resources out on the web do not opt-in using CORS headers and so client side only web apps like a podcast player or a feed reader aren't doable.

One hack-y way around this I've found is to use YQL as a CORS proxy. YQL applies the CORS header to all its responses and among its features it allows a caller to request an arbitrary XML, HTML, or JSON resource. So my network helper script first attempts to access a URI directly using XDomainRequest if that exists and XMLHttpRequest otherwise. If that fails it then tries to use XDR or XHR to access the URI via YQL. I wrap my URIs in the following manner, where type is either "html", "xml", or "json":

        yqlRequest = function(uri, method, type, onComplete, onError) {
            var yqlUri = "http://query.yahooapis.com/v1/public/yql?q=" + 
                encodeURIComponent("SELECT * FROM " + type + ' where url="' + encodeURIComponent(uri) + '"');

            if (type == "html") {
                yqlUri += encodeURIComponent(" and xpath='/*'");
            }
            else if (type == "json") {
                yqlUri += "&callback=&format=json";
            }
            ...

This also means I can get JSON data itself without having to go through JSONP.

As a professional URI aficionado I deal with various levels of ignorance on URI percent-encoding (aka URI encoding, or URL escaping).

Getting into the more subtle levels of URI percent-encoding ignorance, folks try to apply their knowledge of percent-encoding to URIs as a whole producing the concepts escaped URIs and unescaped URIs. However there are no such things - URIs themselves aren't percent-encoded or decoded but rather contain characters that are percent-encoded or decoded. Applying percent-encoding or decoding to a URI as a whole produces a new and non-equivalent URI.

Instead of lingering on the incorrect concepts we'll just cover the correct ones: there's raw unencoded data, non-normal form URIs and normal form URIs. For example:

1. http://example.com/%74%68%65%3F%70%61%74%68?query
2. http://example.com/the%3Fpath?query
3. "http", "example.com", "the?path", "query"

In the above (A) is not an 'encoded URI' but rather a non-normal form URI. The characters of 'the' and 'path' are percent-encoded but as unreserved characters specific in the RFC should not be encoded. In the normal form of the URI (B) the characters are decoded. But (B) is not a 'decoded URI' -- it still has an encoded '?' in it because that's a reserved character which by the RFC holds different meaning when appearing decoded versus encoded. Specifically in this case, it appears encoded which means it is data -- a literal '?' that appears as part of the path segment. This is as opposed to the decoded '?' that appears in the URI which is not part of the path but rather the delimiter to the query.

Usually when developers talk about decoding the URI what they really want is the raw data from the URI. The raw decoded data is (C) above. The only thing to note beyond what's covered already is that to obtain the decoded data one must parse the URI before percent decoding all percent-encoded octets.

Of course the exception here is when a URI is the raw data. In this case you must percent-encode the URI to have it appear in another URI. More on percent-encoding while constructing URIs later.

Sarah and I have been enjoying Glitch for a while now. Reviews are usually positive although occasionally biting (but mostly accurate).

I enjoy Glitch as a game of exploration: exploring the game's lands with hidden and secret rooms, and exploring the games skills and game mechanics. The issue with my enjoyment coming from exploration is that after I've explored all streets and learned all skills I've got nothing left to do. But I've found that even after that I can have fun writing client side JavaScript against Glitch's web APIs making tools (I work on the Glitch Helperator) for use in Glitch. And on a semi-regular basis they add new features reviving my interest in the game itself.

As a professional URI aficionado I deal with various levels of ignorance on URI percent-encoding (aka URI encoding, or URL escaping).

Worse than the lame blog comments hating on percent-encoding is the shipping code which can do actual damage. In one very large project I won't name, I've fixed code that decodes all percent-encoded octets in a URI in order to get rid of pesky percents before calling ShellExecute. An unnamed developer with similar intent but clearly much craftier did the same thing in a loop until the string's length stopped changing. As it turns out percent-encoding serves a purpose and can't just be removed arbitrarily.

Percent-encoding exists so that one can represent data in a URI that would otherwise not be allowed or would be interpretted as a delimiter instead of data. For example, the space character (U+0020) is not allowed in a URI and so must be percent-encoded in order to appear in a URI:

1. http://example.com/the%20path/
2. http://example.com/the path/

In the above the first is a valid URI while the second is not valid since a space appears directly in the URI. Depending on the context and the code through which the wannabe URI is run one may get unexpected failure.

For an additional example, the question mark delimits the path from the query. If one wanted the question mark to appear as part of the path rather than delimit the path from the query, it must be percent-encoded:

1. http://example.com/foo%3Fbar
2. http://example.com/foo?bar

In the second, the question mark appears plainly and so delimits the path "/foo" from the query "bar". And in the first, the querstion mark is percent-encoded and so the path is "/foo%3Fbar".

As a professional URI aficionado I deal with various levels of ignorance on URI percent-encoding (aka URI encoding, or URL escaping). The basest ignorance is with respect to the mere existence of percent-encoding. Percents in URIs are special: they always represent the start of a percent-encoded octet. That is to say, a percent is always followed by two hex digits that represents a value between 0 and 255 and doesn't show up in a URI otherwise.

The IPv6 textual syntax for scoped addresses uses the '%' to delimit the zone ID from the rest of the address. When it came time to define how to represent scoped IPv6 addresses in URIs there were two camps: Folks who wanted to use the IPv6 format as is in the URI, and those who wanted to encode or replace the '%' with a different character. The resulting thread was more lively than what shows up on the IETF URI discussion mailing list. Ultimately we went with a percent-encoded '%' which means the percent maintains its special status and singular purpose.

For my GeolocMock weekend project I intended to use the Bing Maps API to display a map in a WebBrowser control and allow the user to interact with that to select a location to be consumed by my application. Getting my .NET code to talk to the JavaScript in the WebBrowser control was surprisingly easy.

To have .NET execute JavaScript code you can use the InvokeScript method passing the name of the JavaScript function to execute and an object array of parameters to pass:

this.webBrowser2.Document.InvokeScript("onLocationStateChanged",
   new object[] {
      latitudeTextBoxText,
      longitudeTextBoxText,
      altitudeTextBoxText,
      uncertaintyTextBoxText
   });

The other direction, having JavaScript call into .NET is slightly more complicated but still pretty easy as far as language interop goes. The first step is to mark your assembly as ComVisible so that it can interact with JavaScript via COM. VS had already added a ComVisible declaration to my project I just had to change the value to true.

[assembly: ComVisible(true)]

Next set ObjectForScripting attribute to the object you want to expose to JavaScript.

this.webBrowser2.ObjectForScripting = this.locationState;

Now that object is exposed as window.external in JavaScript and you can call methods on it.

window.external.Set(lat, long, alt, gUncert);

However you don't seem to be able to test for the existence of methods off of it. For example the following JavaScript generates an exception for me even though I have a Set method:

if (window.external && window.external.Set) {

Working on GeolocMock it took me a bit to realize why my HTML could use the W3C Geolocation API in IE9 but not in my WebBrowser control in my .NET application. Eventually I realized that I was getting the wrong IE doc mode. Reading this old More IE8 Extensibility Improvements IE blog post from the IE blog I found the issue is that for app compat the WebOC picks older doc modes but an app hosting the WebOC can set a regkey to get different doc modes. The IE9 mode isn't listed in that article but I took a guess based on the values there and the decimal value 9999 gets my app IE9 mode. The following is the code I run in my application to set its regkey so that my app can get the IE9 doc mode and use the geolocation API.

        static private void UseIE9DocMode()
        {
            RegistryKey key = null;
            try
            {
                key = Registry.CurrentUser.OpenSubKey("Software\\Microsoft\\Internet Explorer\\Main\\FeatureControl\\FEATURE_BROWSER_EMULATION", true);
            }
            catch (Exception)
            {
                key = Registry.CurrentUser.CreateSubKey("Software\\Microsoft\\Internet Explorer\\Main\\FeatureControl\\FEATURE_BROWSER_EMULATION");
            }
            key.SetValue(System.Diagnostics.Process.GetCurrentProcess().MainModule.ModuleName, 9999, RegistryValueKind.DWord);
            key.Close();
        }

I've made GeolocMock. If your PC has no geolocation devices, IE9 uses a webservice to determine your location. GeolocMock uses FiddlerCore to intercept the response from the webservice and allows the user to replace the location in the response with another. This was a fun weekend project in order to play with FiddlerCore, the W3C Geoloc APIs in IE9, hosting the IE9 WebOC in a .NET app, and the Bing Maps APIs.