syntax page 2 - Dave's Blog

QuickBase Formula Pretty Printer and Syntax Highlighter

2008 Oct 5, 9:17

Sarah asked me if I knew of a syntax highlighter for the QuickBase formula language which she uses at work. I couldn't find one but thought it might be fun to make a QuickBase Formula syntax highlighter based on the QuickBase help's description of the formula syntax. Thankfully the language is relatively simple since my skills with ANTLR, the parser generator, are rusty now and I've only used it previously for personal projects (like Javaish, the ridiculous Java based shell idea I had).

With the help of some great ANTLR examples and an ANTLR cheat sheet I was able to come up with the grammar that parses the QuickBase Formula syntax and prints out the same formula marked up with HTML SPAN tags and various CSS classes. ANTLR produces the parser in Java which I wrapped up in an applet, put in a jar, and embedded in an HTML page. The script in that page runs user input through the applet's parser and sticks the output at the bottom of the page with appropriate CSS rules to highlight and print the formula in a pretty fashion.

What I learned:

• I didn't realize that Java applets are easy to use via script in an HTML page. In the JavaScript I can simply refer to publicly exposed methods on the applet and run JavaScript strings through them. It makes for a great combination: do the heavy coding in Java and do the UI in HTML. I may end up doing this again in the future.
• I love ANTLRWorks, the ANTLR IDE, that didn't exist the last time I used ANTLR. It tells you about issues with your grammar as you create it, lets you easily debug the grammar running it forwards and backwards, display parse trees, and other useful things.

java technical programming quickbase language antlr antlrworks

URI Fragment Info Roundup

2008 Apr 21, 11:53

Information about URI Fragments, the portion of URIs that follow the '#' at the end and that are used to navigate within a document, is scattered throughout various documents which I usually have to hunt down. Instead I'll link to them all here.

Definitions. Fragments are defined in the URI RFC which states that they're used to identify a secondary resource that is related to the primary resource identified by the URI as a subset of the primary, a view of the primary, or some other resource described by the primary. The interpretation of a fragment is based on the mime type of the primary resource. Tim Berners-Lee notes that determining fragment meaning from mime type is a problem because a single URI may contain a single fragment, however over HTTP a single URI can result in the same logical resource represented in different mime types. So there's one fragment but multiple mime types and so multiple interpretations of the one fragment. The URI RFC says that if an author has a single resource available in multiple mime types then the author must ensure that the various representations of a single resource must all resolve fragments to the same logical secondary resource. Depending on which mime types you're dealing with this is either not easy or not possible.

HTTP. In HTTP when URIs are used, the fragment is not included. The General Syntax section of the HTTP standard says it uses the definitions of 'URI-reference' (which includes the fragment), 'absoluteURI', and 'relativeURI' (which don't include the fragment) from the URI RFC. However, the 'URI-reference' term doesn't actually appear in the BNF for the protocol. Accordingly the headers like 'Request-URI', 'Content-Location', 'Location', and 'Referer' which include URIs are defined with 'absoluteURI' or 'relativeURI' and don't include the fragment. This is in keeping with the original fragment definition which says that the fragment is used as a view of the original resource and consequently only needed for resolution on the client. Additionally, the URI RFC explicitly notes that not including the fragment is a privacy feature such that page authors won't be able to stop clients from viewing whatever fragments the client chooses. This seems like an odd claim given that if the author wanted to selectively restrict access to portions of documents there are other options for them like breaking out the parts of a single resource to which the author wishes to restrict access into separate resources.

HTML. In HTML, the HTML mime type RFC defines HTML's fragment use which consists of fragments referring to elements with a corresponding 'id' attribute or one of a particular set of elements with a corresponding 'name' attribute. The HTML spec discusses fragment use additionally noting that the names and ids must be unique in the document and that they must consist of only US-ASCII characters. The ID and NAME attributes are further restricted in section 6 to only consist of alphanumerics, the hyphen, period, colon, and underscore. This is a subset of the characters allowed in the URI fragment so no encoding is discussed since technically its not needed. However, practically speaking, browsers like FireFox and Internet Explorer allow for names and ids containing characters outside of the defined set including characters that must be percent-encoded to appear in a URI fragment. The interpretation of percent-encoded characters in fragments for HTML documents is not consistent across browsers (or in some cases within the same browser) especially for the percent-encoded percent.

Text. Text/plain recently got a fragment definition that allows fragments to refer to particular lines or characters within a text document. The scheme no longer includes regular expressions, which disappointed me at first, but in retrospect is probably good idea for increasing the adoption of this fragment scheme and for avoiding the potential for ubiquitous DoS via regex. One of the authors also notes this on his blog. I look forward to the day when this scheme is widely implemented.

XML. XML has the XPointer framework to define its fragment structure as noted by the XML mime type definition. XPointer consists of a general scheme that contains subschemes that identify a subset of an XML document. Its too bad such a thing wasn't adopted for URI fragments in general to solve the problem of a single resource with multiple mime type representations. I wrote more about XPointer when I worked on hacking XPointer into IE.

SVG and MPEG. Through the Media Fragments Working Group I found a couple more fragment scheme definitions. SVG's fragment scheme is defined in the SVG documentation and looks similar to XML's. MPEG has one defined but I could only find it as an ISO document "Text of ISO/IEC FCD 21000-17 MPEG-12 FID" and not as an RFC which is a little disturbing.

AJAX. AJAX websites have used fragments as an escape hatch for two issues that I've seen. The first is getting a unique URL for versions of a page that are produced on the client by script. The fragment may be changed by script without forcing the page to reload. This goes outside the rules of the standards by using HTML fragments in a fashion not called out by the HTML spec. but it does seem to be inline with the spirit of the fragment in that it is a subview of the original resource and interpretted client side. The other hack-ier use of the fragment in AJAX is for cross domain communication. The basic idea is that different frames or windows may not communicate in normal fashions if they have different domains but they can view each other's URLs and accordingly can change their own fragments in order to send a message out to those who know where to look. IMO this is not inline with the spirit of the fragment but is rather a cool hack.

xml text ajax technical url boring uri fragment rfc

IPv6 Roundup: Address Syntax on Windows

2008 Jan 9, 11:34

IPv6 address syntax consists of 8 groupings of colon delimited 16-bit hex values making up the 128-bit address. An optional double colon can replace any consecutive sequence of 0 valued hex values. For example the following is a valid IPv6 address: fe80::2c02:db79

Some IPv6 addresses aren't global and in those cases need a scope ID to describe their context. These get a '%' followed by the scope ID. For example the previous example with a scope ID of '8' would be: fe80::2c02:db79%8

IPv6 addresses in URIs may appear in the host section of a URI as long as they're enclosed by square brackets. For example: http://[fe80::2c02:db79]/. The RFC explicitly notes that there isn't a way to add a scope ID to the IPv6 address in a URI. However a draft document describes adding scope IDs to IPv6 addresses in URIs. The draft document uses the IPvFuture production from the URI RFC with a 'v1' to add a new hostname syntax and a '+' instead of a '%' for delimiting the scope id. For example: http://[v1.fe80::2c02:db79+8]/. However, this is still a draft document, not a final standard, and I don't know of any system that works this way.

In Windows XPSP2 the IPv6 stack is available but disabled by default. To enable the IPv6 stack, at a command prompt run 'netsh interface ipv6 install'. In Vista IPv6 is the on by default and cannot be turned off, while the IPv4 stack is optional and may be turned off by a command similar to the previous.

Once you have IPv6 on in your OS you can turn on IPv6 for IIS6 or just use IIS7. The address ::1 refers to the local machine.

In some places in Windows like UNC paths, IPv6 addresses aren't allowed. In those cases you can use a Vista DNS IPv6 hack that lives in the OS name resolution stack that transforms particularly crafted names into IPv6 addresses. Take your IPv6 address, replace the ':'s with '-'s and the '%' with an 's' and then append '.ipv6-literal.net' to the end. For example: fe80--2c02-db79s8.ipv6-literal.net. That name will resolve to the same example I've been using in Vista. This transformation occurs inside the system's local name resolution stack so no DNS servers are involved, although Microsoft does own the ipv6-literal.net domain name.

MSDN describes IPv6 addresses in URIs in Windows and I've described IPv6 addresses in URIs in IE7. File URIs in IE7 don't support IPv6 addresses. If you want to put a scope ID in a URI in IE7 you use a '%25' to delimit the scope ID and due to a bug you must have at least two digits in your scope ID. So, to take the previous example: http://[fe80::2c02:db79%2508]/. Note that its 08 rather than just 8.

roundup ip windows ipv6 technical microsoft boring syntax

Which which - Batch File Hackiness

2007 Aug 9, 5:41To satisfy my hands which have already learned to type *nix commands I like to install Win32 versions of common GNU utilities. Unfortunately, the which command is a rather literal port and requires you to enter the entire name of the command for which you're looking. That is 'which which' won't find itself but 'which which.exe' will. This makes this almost useless for me so I thought to write my own as a batch file. I had learned about a few goodies available in cmd.exe that I thought would make this an easy task. It turned out to be more difficult than I thought.

for /F "usebackq tokens=*" %%a in ( `"echo %PATH:;=& echo %"` ) do (
    for /F "usebackq tokens=*" %%b in ( `"echo %PATHEXT:;=& echo %"` ) do (
        if exist "%%a"\%1%%b (
            for  %%c in ( "%%a"\%1%%b ) do (
                echo %%~fc
            )
        )
    )
)

The environment variables PATH and PATHEXT hold the list of paths to search through to find commands, and the extensions of files that should be run as commands respectively. The 'for /F "usebackq tokens=*" %%a in (...) do (...)' runs the 'do' portion with %%a sequentially taking on the value of every line in the 'in' portion. That's nice, but PATH and PATHEXT don't have their elements on different lines and I don't know of a way to escape a newline character to appear in a batch file. In order to get the PATH and PATHEXT's elements onto different lines I used the %ENV:a=b% syntax which replaces occurrences of a with b in the value of ENV. I replaced the ';' delimiter with the text '& echo ' which means %PATHEXT:;=& echo% evaluates to something like "echo .COM& echo .EXE& echo .BAT& ...". I have to put the whole expression in double quotes in order to escape the '&' for appearing in the batch file. The usebackq and the backwards quotes means that the backquoted string should be replaced with the output of the execution of its content. So in that fashion I'm able to get each element of the env. variable onto new lines. The rest is pretty straight forward.

Also, it supports wildcards:

C:\Users\davris>which.cmd *hi*
C:\Windows\System32\GRAPHICS.COM
C:\Windows\System32\SearchIndexer.exe
D:\bin\which.exe
D:\bin\which.cmd

which cmd technical batch for

New XSLT - IE7 XML Source View Upgrade Part 2

2007 May 11, 8:55Last time, I had written some resource tools to allow me to view and modify Windows module resources in my ultimate and noble quest to implement the XML content-type fragment in IE7. Using the resource tools I found that MSXML3.DLL isn't signed and that I can replace the XSLT embedded resource with my own, which is great news and means I could continue in my endevour. In the following I discuss how I came up with this replacement for IE7's XML source view.

At first I thought I could just modify the existing XSLT but it turns out that it isn't exactly an XSLT, rather its an IE5 XSL. I tried using the XSL to XSLT converter linked to on MSDN, however the resulting document still requires manual modification. But I didn't want to muck about in their weird language and I figured I could write my own XSLT faster than I could figure out how theirs worked.

I began work on the new XSLT and found it relatively easy to produce. First I got indenting working with all the XML nodes represented appropriately and different CSS classes attached to them to make it easy to do syntax highlighting. Next I added in some javascript to allow for closing and opening of elements. At this point my XSLT had the same features as the original XSL.

Next was the XML mimetype fragment which uses XPointer, a framework around various different schemes for naming parts of an XML document. I focused on the XPointer scheme which is an extended version of XPath. So I named my first task as getting XPaths working. Thankfully javascript running in the HTML document produced by running my XSLT on an XML document has access to the original XML document object via the document.XMLDocument property. From this this I can execute XPaths, however there's no builtin way to map from the XML nodes selected by the XPath to the HTML elements that I produced to represent them. So I created a recursive javascript function and XSLT named-template that both produce the same unique strings based on an XML node's position in the document. For instance 'a3-e2-e' is the name produced for the 3rd attribute of the second element of the root element of the XML document. When producing the HTML for an XML node, I add an 'id' attribute to the HTML with the unique string of the XML node. Then in javascript when I execute an XPath I can discover the unique string of each node in the selected set and map each of them to their corresponding positions in the HTML.

With the hard part out of the way I changed the onload to get the fragment of the URI of the current document, interpret it as an XPath and highlight and navigate to the selected nodes. I also added an interactive floating bar from which you can enter your own XPaths and do the same. On a related note, I found that when accessing XML files via the file URI scheme the fragment is stripped off and not available to the javascript.

The next steps are of course to actually implement XPointer framework parsing as well as the limited number of schemes that the XPointer framework specifies.xml xpointer msxml res xpath xslt resource ie7 technical browser ie xsl

Resource Tools - IE7 XML Source View Upgrade Part 1

2007 May 9, 4:15I read about text/xml URI fragment resolution a few months ago. I was interested to find another kind of fragment reference other than the text/html URI fragment but of course I didn't find an implementation in IE, Firefox, or Opera. I decided to see how much work would be required to implement this in IE.

In IE and Firefox when you open an XML file that doesn't have an XML stylesheet the XML source is rendered with syntax highlighting. In IE I also noticed that the gold bar appears when you open an XML file off of your local machine. To me this suggested that the XML source was being rendered as HTML which I assumed was produced by running an XSLT on the source XML file. If so, I figured I could modify the XSLT to implement text/xml URI fragments. I ran FileMon to see if iexplore.exe loaded an XSLT file when opening an XML file. Only the XML file and MSXML3.DLL were opened and no XSLTs were loaded as files. My next hope for modifying the XSLT was if it existed as a resource in MSXML3.DLL. I did a findstr on the DLL for SCRIPT and found an XSLT so I decided to check for resources in MSXML3.DLL. Unfortunately my previous resource viewer didn't work correctly so I decided to write my own.

I created resource tools to view and modify resources in Windows modules. The viewer outputs HTML with links to the individual resources of a module using the res URI scheme that's built into IE. The modifier is a simple command line tool that replaces or adds one resource at a time to a module.

Using these tools I found that the XSLT was stored as a resource in MSXML3.DLL. I'll talk more about the existing XSLT and the one I replaced it with next time.resource technical xml msxml res xslt xsl

Delicious shortcut tag

2007 Apr 8, 3:05Shortcut Tag?
I just saw this on another user's delicious links: a link to ESV search that's tagged with, among other things, "shortcut:esv". When viewed on del.icio.us there's a text box that lets you search using that link. I hadn't seen this before, but it seems pretty cool and I'm surprised I hadn't seen it previously. A delicious post with such a tag ends up looking like the following: I tried searching for information on this and I've found other delicious users doing the same thing, but nothing about the tag itself. If you know any information especially official information from del.icio.us itself please post links in reply to this post. So without further preface here's what I've learned about the del.icio.us shortcut tag.

How-to
To get a search box in your del.icio.us links make a post that satisfies the following requirements:

1. One of the tags must begin with the text 'shortcut:'. You can have more text following that in the tag if you like but it must at least start with 'shortcut:'.
2. The 'url' you post must be a shortcut url rather than an actual URL. It must contain a '%s' with a lowercase 's'. When you enter text into the textbox on the del.icio.us page the text will replace the '%s' after being percent-encoded. For example 'http://www.google.com/search?hl=en&q=%s' is the shortcut url for Google and if you type 'foo bar' into the textbox the URI you will navigate to would be 'http://www.google.com/search?hl=en&q=foo%20bar'.

Complaints
This is neat but I do have a few complaints:

1. The text from the textbox is percent-encoded before replacing the '%s'. Most sites use application/x-www-form-urlencoded which encodes spaces as '+' rather than '%20'.
2. The shortcut url format seems to be taken from Mozilla's Firefox Custom Keywords. Its a shame it wasn't based on something more adaptable like the OpenSearch URL template syntax.
3. A '%s' in the url means technically what you're submitting to del.icio.us isn't a URI as defined by the standard.
4. Allowing text after 'shortcut:' means you can't look at all of a user's shortcut using this tag.

The next step is to create a tool to sync my IE7 search providers with my shortcuts saved to delicious...technical howto tagging tool tag delicious