string - Dave's Blog

Search
My timeline on Mastodon

Changing the User Agent string in UWP WebView

2018 Oct 23, 9:32

There's no perfect way to change the user agent string for the UWP WebView (x-ms-webview in HTML, Windows.UI.Xaml.Controls.WebView in XAML, and Windows.Web.UI.Interop.WebViewControl in Win32) but there are two imperfect methods folks end up using.

The first is to call UrlMkSetSessionOption. This is an old public API that allows you to configure various arcane options including one that is the default user agent string for requests running through urlmon. This API is allowed by the Microsoft Store for UWP apps. The change it applies is process wide which has two potential drawbacks. If you want to be able to have different UA strings set for different requests from a WebView that's not really possible with this solution. The other drawback is if you're using out of process WebView, you need to ensure you're calling into UrlMkSetSessionOption in the WebView's process. You'll need to write third party WinRT that calls UrlMkSetSessionOption, create the out of proc WebView, navigate it to some trusted local page, use AddWebAllowedObject or provide that URI WinRT access, and call into your third party WinRT. You'll need to do that for any new WebView process you create.

The second less generally applicable solution is to use NavigateWithHttpRequestMessage and set the User-Agent HTTP header. In this case you get to control the scope of the user agent string changes but has the limitations that not all sub resource downloads will use this user agent string and for navigations you don't initiate you have to manually intercept and re-request being careful to transfer over all POST body state and HTTP headers correctly. That last part is not actually possible for iframes.

PermalinkCommentsuser-agent uwp webview

Win10 UWP WebView AddWebAllowedObject details

2017 Sep 4, 3:09

The x-ms-webview HTML element has the void addWebAllowedObject(string name, any value) method and the webview XAML element has the void AddWebAllowedObject(String name, Object value) method. The object parameter is projected into the webview’s top-level HTML document’s script engine as a new property on the global object with property name set to the name parameter. It is not injected into the current document but rather it is projected during initialization of the next top-level HTML document to which the webview navigates.

Lifetime

If AddWebAllowedObject is called during a NavigationStarting event handler the object will be injected into the document resulting from the navigation corresponding to that event.

If AddWebAllowedObject is called outside of the NavigationStarting event handler it will apply to the navigation corresponding to the next explicit navigate method called on the webview or the navigation corresponding to the next NavigationStarting event handler that fires, whichever comes first.

To avoid this potential race, you should use AddWebAllowedObject in one of two ways: 1. During a NavigationStarting event handler, 2. Before calling a Navigate method and without returning to the main loop.

If called both before calling a navigate method and in the NavigationStarting event handler then the result is the aggregate of all those calls.

If called multiple times for the same document with the same name the last call wins and the previous are silently ignored.

If AddWebAllowedObject is called for a navigation and that navigation fails or redirects to a different URI, the AddWebAllowedObject call is silently ignored.

After successfully adding an object to a document, the object will no longer be projected once a navigation to a new document occurs.

WinRT access

If AddWebAllowedObject is called for a document with All WinRT access then projection will succeed and the object will be added.

If AddWebAllowedObject is called for a document which has a URI which has no declared WinRT access via ApplicationContentUriRules then Allow for web only WinRT access is given to that document.

If the document has Allow for web only WinRT access then projection will succeed only if the object’s runtimeclass has the Windows.Foundation.Metadata.AllowForWeb metadata attribute.

Object requirements

The object must implement the IAgileObject interface. Because the XAML and HTML webview elements run on ASTA view threads and the webview’s content’s JavaScript thread runs on another ASTA thread a developer should not create their non-agile runtimeclass on the view thread. To encourage end developers to do this correctly we require the object implements IAgileObject.

Property name

The name parameter must be a valid JavaScript property name, otherwise the call will fail silently. If the name is already a property name on the global object, that property is overwritten if the property is configurable. Non-configurable properties on the global object are not overwritten and the AddWebAllowedObject call fails silently. On success, the projected property is writable, configurable, and enumerable.

Errors

Some errors as described above fail silently. Other issues, such as lack of IAgileObject or lack of the AllowForWeb attribute result in an error in the JavaScript developer console.

PermalinkComments

Application Content URI Rules wildcard syntax

2017 May 31, 4:48

Application Content URI Rules (ACUR from now on) defines the bounds of the web that make up the Microsoft Store application. Package content via the ms-appx URI scheme is automatically considered part of the app. But if you have content on the web via http or https you can use ACUR to declare to Windows that those URIs are also part of your application. When your app navigates to URIs on the web those URIs will be matched against the ACUR to determine if they are part of your app or not. The documentation for how matching is done on the wildcard URIs in the ACUR Rule elements is not very helpful on MSDN so here are some notes.

Rules

You can have up to 100 Rule XML elements per ApplicationContentUriRules element. Each has a Match attribute that can be up to 2084 characters long. The content of the Match attribute is parsed with CreateUri and when matching against URIs on the web additional wildcard processing is performed. I’ll call the URI from the ACUR Rule the rule URI and the URI we compare it to found during app navigation the navigation URI.

The rule URI is matched to a navigation URI by URI component: scheme, username, password, host, port, path, query, and fragment. If a component does not exist on the rule URI then it matches any value of that component in the navigation URI. For example, a rule URI with no fragment will match a navigation URI with no fragment, with an empty string fragment, or a fragment with any value in it.

Asterisk

Each component except the port may have up to 8 asterisks. Two asterisks in a row counts as an escape and will match 1 literal asterisk. For scheme, username, password, query and fragment the asterisk matches whatever it can within the component.

Host

For the host, if the host consists of exactly one single asterisk then it matches anything. Otherwise an asterisk in a host only matches within its domain name label. For example, http://*.example.com will match http://a.example.com/ but not http://b.a.example.com/ or http://example.com/. And http://*/ will match http://example.com, http://a.example.com/, and http://b.a.example.com/. However the Store places restrictions on submitting apps that use the http://* rule or rules with an asterisk in the second effective domain name label. For example, http://*.com is also restricted for Store submission.

Path

For the path, an asterisk matches within the path segment. For example, http://example.com/a/*/c will match http://example.com/a/b/c and http://example.com/a//c but not http://example.com/a/b/b/c or http://example.com/a/c

Additionally for the path, if the path ends with a slash then it matches any path that starts with that same path. For example, http://example.com/a/ will match http://example.com/a/b and http://example.com/a/b/c/d/e/, but not http://example.com/b/.

If the path doesn’t end with a slash then there is no suffix matching performed. For example, http://example.com/a will match only http://example.com/a and no URIs with a different path.

As a part of parsing the rule URI and the navigation URI, CreateUri will perform URI normalization and so the hostname and scheme will be made lower case (casing matters in all other parts of the URI and case sensitive comparisons will be performed), IDN normalization will be performed, ‘.’ and ‘..’ path segments will be resolved and other normalizations as described in the CreateUri documentation.

PermalinkCommentsapplication-content-uri-rules programming windows windows-store

Cdb/Windbg Commands for Runtime Patching

2016 Feb 8, 1:47

You can use conditional breakpoints and debugging commands in windbg and cdb that together can amount to effectively patching a binary at runtime. This can be useful if you have symbols but you can't easily rebuild the binary. Or if the patch is small and the binary requires a great deal of time to rebuild.

Skipping code

If you want to skip a chunk of code you can set a breakpoint at the start address of the code to skip and set the breakpoint's command to change the instruction pointer register to point to the address at the end of the code to skip and go. Voila you're skipping over that code now. For example:

bp 0x6dd6879b "r @eip=0x6dd687c3 ; g"

Changing parameters

You may want to modify parameters or variables and this is simple of course. In the following example a conditional breakpoint ANDs out a bit from dwFlags. Now when we run its as if no one is passing in that flag.

bp wiwi!RelativeCrack "?? dwFlags &= 0xFDFFFFFF;g"

Slightly more difficult is to modify string values. If the new string length is the same size or smaller than the previous, you may be able to modify the string value in place. But if the string is longer or the string memory isn't writable, you'll need a new chunk of memory into which to write your new string. You can use .dvalloc to allocate some memory and ezu to write a string into the newly allocated memory. In the following example I then overwrite the register containing the parameter I want to modify:

.dvalloc 100
ezu 000002a9`d4eb0000 "mfcore.dll"
r rcx = 000002a9`d4eb0000

Calling functions

You can also use .call to actually make new calls to methods or functions. Read more about that on the Old New Thing: Stupid debugger tricks: Calling functions and methods. Again, all of this can be used in a breakpoint command to effectively patch a binary.

PermalinkCommentscdb debug technical windbg

Unicode Clock

2016 Jan 24, 2:00

I've made a Unicode Clock in JavaScript.

Unicode has code points for all 30 minute increments of clock faces. This is a simple project to display the one closest to the current time written in JavaScript.

Because the code points are all above 0xFFFF, I make use of some ES6 additions. I use the \u{XXXXXX} style escape sequence since the old style JavaScript escape sequence \uXXXX only supports code points up to 0xFFFF. I also use the method String.codePointAt rather than String.charCodeAt because the code points larger than 0xFFFF are represented in JavaScript strings using surrogate pairs and charCodeAt gives the surrogate value rather than codePointAt which gives the code point represented by the pair of surrogates.

"🕛".codePointAt(0)
128347
"🕛".charCodeAt(0)
55357

🕐🕑🕒🕓🕔🕕🕖🕗🕘🕙🕚🕛🕜🕝🕞🕟🕠🕡🕢🕣🕤🕥🕦🕧

The ordering of the code points does not make it simple to do this. I initially guessed the first code point in the range would be 12:00 followed by 12:30, 1:00 and so on. But actually 1:00 is first followed by all the on the hour times then all the half hour times.

PermalinkCommentsjavascript Unicode

JavaScript Types and WinRT Types

2016 Jan 21, 5:35

MSDN covers the topic of JavaScript and WinRT type conversions provided by Chakra (JavaScript Representation of Windows Runtime Types and Considerations when Using the Windows Runtime API), but for the questions I get about it I’ll try to lay out some specifics of that discussion more plainly. I’ve made a TL;DR JavaScript types and WinRT types summary table.

WinRT Conversion JavaScript
Struct ↔️ JavaScript object with matching property names
Class or interface instance JavaScript object with matching property names
Windows.Foundation.Collections.IPropertySet JavaScript object with arbitrary property names
Any DOM object

Chakra, the JavaScript engine powering the Edge browser and JavaScript Windows Store apps, does the work to project WinRT into JavaScript. It is responsible for, among other things, converting back and forth between JavaScript types and WinRT types. Some basics are intuitive, like a JavaScript string is converted back and forth with WinRT’s string representation. For other basic types check out the MSDN links at the top of the page. For structs, interface instances, class instances, and objects things are more complicated.

A struct, class instance, or interface instance in WinRT is projected into JavaScript as a JavaScript object with corresponding property names and values. This JavaScript object representation of a WinRT type can be passed into other WinRT APIs that take the same underlying type as a parameter. This JavaScript object is special in that Chakra keeps a reference to the underlying WinRT object and so it can be reused with other WinRT APIs.

However, if you start with plain JavaScript objects and want to interact with WinRT APIs that take non-basic WinRT types, your options are less plentiful. You can use a plain JavaScript object as a WinRT struct, so long as the property names on the JavaScript object match the WinRT struct’s. Chakra will implicitly create an instance of the WinRT struct for you when you call a WinRT method that takes that WinRT struct as a parameter and fill in the WinRT struct’s values with the values from the corresponding properties on your JavaScript object.

// C# WinRT component
public struct ExampleStruct
{
public string String;
public int Int;
}

public sealed class ExampleStructContainer
{
ExampleStruct value;
public void Set(ExampleStruct value)
{
this.value = value;
}

public ExampleStruct Get()
{
return this.value;
}
}

// JS code
var structContainer = new ExampleWinRTComponent.ExampleNamespace.ExampleStructContainer();
structContainer.set({ string: "abc", int: 123 });
console.log("structContainer.get(): " + JSON.stringify(structContainer.get()));
// structContainer.get(): {"string":"abc","int":123}

You cannot have a plain JavaScript object and use it as a WinRT class instance or WinRT interface instance. Chakra does not provide such a conversion even with ES6 classes.

You cannot take a JavaScript object with arbitrary property names that are unknown at compile time and don’t correspond to a specific WinRT struct and pass that into a WinRT method. If you need to do this, you have to write additional JavaScript code to explicitly convert your arbitrary JavaScript object into an array of property name and value pairs or something else that could be represented in WinRT.

However, the other direction you can do. An instance of a Windows.Foundation.Collections.IPropertySet implementation in WinRT is projected into JavaScript as a JavaScript object with property names and values corresponding to the key and value pairs in the IPropertySet. In this way you can project a WinRT object as a JavaScript object with arbitrary property names and types. But again, the reverse is not possible. Chakra will not convert an arbitrary JavaScript object into an IPropertySet.

// C# WinRT component
public sealed class PropertySetContainer
{
private Windows.Foundation.Collections.IPropertySet otherValue = null;

public Windows.Foundation.Collections.IPropertySet other
{
get
{
return otherValue;
}
set
{
otherValue = value;
}
}
}

public sealed class PropertySet : Windows.Foundation.Collections.IPropertySet
{
private IDictionary map = new Dictionary();

public PropertySet()
{
map.Add("abc", "def");
map.Add("ghi", "jkl");
map.Add("mno", "pqr");
}
// ... rest of PropertySet implementation is simple wrapper around the map member.


// JS code
var propertySet = new ExampleWinRTComponent.ExampleNamespace.PropertySet();
console.log("propertySet: " + JSON.stringify(propertySet));
// propertySet: {"abc":"def","ghi":"jkl","mno":"pqr"}

var propertySetContainer = new ExampleWinRTComponent.ExampleNamespace.PropertySetContainer();
propertySetContainer.other = propertySet;
console.log("propertySetContainer.other: " + JSON.stringify(propertySetContainer.other));
// propertySetContainer.other: {"abc":"def","ghi":"jkl","mno":"pqr"}

try {
propertySetContainer.other = { "123": "456", "789": "012" };
}
catch (e) {
console.error("Error setting propertySetContainer.other: " + e);
// Error setting propertySetContainer.other: TypeError: Type mismatch
}

There’s also no way to implicitly convert a DOM object into a WinRT type. If you want to write third party WinRT code that interacts with the DOM, you must do so indirectly and explicitly in JavaScript code that is interacting with your third party WinRT. You’ll have to extract the information you want from your DOM objects to pass into WinRT methods and similarly have to pass messages out from WinRT that say what actions the JavaScript should perform on the DOM.

PermalinkCommentschakra development javascript winrt

Retweet of velartrill

2015 Jun 29, 7:23
the year is "2027". strings are the only remaining data type
PermalinkComments

randomdross: UA string with a shortlink to moar UA string

2015 Jan 21, 9:12
David Ross @randomdross :
UA string with a shortlink to moar UA string
PermalinkComments

David_Risney: Learn about ES6 template strings: I didn't know about tagged templates allowing for HTML encoding in template strings

2015 Jan 21, 12:15
David Risney @David_Risney :
Learn about ES6 template strings: http://updates.html5rocks.com/2015/01/ES6-Template-Strings … I didn't know about tagged templates allowing for HTML encoding in template strings
PermalinkComments

location.hash and location.search are bad and they should feel bad

2014 May 22, 9:25
The DOM location interface exposes the HTML document's URI parsed into its properties. However, it is ancient and has problems that bug me but otherwise rarely show up in the real world. Complaining about mostly theoretical issues is why blogging exists, so here goes:
  • The location object's search, hash, and protocol properties are all misnomers that lead to confusion about the correct terms:
    • The 'search' property returns the URI's query property. The query property isn't limited to containing search terms.
    • The 'hash' property returns the URI's fragment property. This one is just named after its delimiter. It should be called the fragment.
    • The 'protocol' property returns the URI's scheme property. A URI's scheme isn't necessarily a protocol. The http URI scheme of course uses the HTTP protocol, but the https URI scheme is the HTTP protocol over SSL/TLS - there is no HTTPS protocol. Similarly for something like mailto - there is no mailto wire protocol.
  • The 'hash' and 'search' location properties both return null in the case that their corresponding URI property doesn't exist or if its the empty string. A URI with no query property and a URI with an empty string query property that are otherwise the same, are not equal URIs and are allowed by HTTP to return different content. Similarly for the fragment. Unless the specific URI scheme defines otherwise, an empty query or hash isn't the same as no query or hash.
But like complaining about the number of minutes in an hour none of this can ever change without huge compat issues on the web. Accordingly I can only give my thanks to Anne van Kesteren and the awesome work on the URL standard moving towards a more sane (but still working practically within the constraints of compat) location object and URI parsing in the browser.
PermalinkComments

Moving PowerShell data into Excel

2013 Aug 15, 10:04
PowerShell nicely includes ConvertTo-CSV and ConvertFrom-CSV which allow you to serialize and deserialize your PowerShell objects to and from CSV. Unfortunately the CSV produced by ConvertTo-CSV is not easily opened by Excel which expects by default different sets of delimiters and such. Looking online you'll find folks who recommend using automation via COM to create a new Excel instance and copy over the data in that fashion. This turns out to be very slow and impractical if you have large sets of data. However you can use automation to open CSV files with not the default set of delimiters. So the following isn't the best but it gets Excel to open a CSV file produced via ConvertTo-CSV and is faster than the other options:
Param([Parameter(Mandatory=$true)][string]$Path);

$excel = New-Object -ComObject Excel.Application

$xlWindows=2
$xlDelimited=1 # 1 = delimited, 2 = fixed width
$xlTextQualifierDoubleQuote=1 # 1= doublt quote, -4142 = no delim, 2 = single quote
$consequitiveDelim = $False;
$tabDelim = $False;
$semicolonDelim = $False;
$commaDelim = $True;
$StartRow=1
$Semicolon=$True

$excel.visible=$true
$excel.workbooks.OpenText($Path,$xlWindows,$StartRow,$xlDelimited,$xlTextQualifierDoubleQuote,$consequitiveDelim,$tabDelim,$semicolonDelim, $commaDelim);
See Workbooks.OpenText documentation for more information.
PermalinkCommentscsv excel powershell programming technical

URI functions in Windows Store Applications

2013 Jul 25, 1:00

Summary

The Modern SDK contains some URI related functionality as do libraries available in particular projection languages. Unfortunately, collectively these APIs do not cover all scenarios in all languages. Specifically, JavaScript and C++ have no URI building APIs, and C++ additionally has no percent-encoding/decoding APIs.
WinRT (JS and C++)
JS Only
C++ Only
.NET Only
Parse
 
Build
Normalize
Equality
 
 
Relative resolution
Encode data for including in URI property
Decode data extracted from URI property
Build Query
Parse Query
The Windows.Foudnation.Uri type is not projected into .NET modern applications. Instead those applications use System.Uri and the platform ensures that it is correctly converted back and forth between Windows.Foundation.Uri as appropriate. Accordingly the column marked WinRT above is applicable to JS and C++ modern applications but not .NET modern applications. The only entries above applicable to .NET are the .NET Only column and the WwwFormUrlDecoder in the bottom left which is available to .NET.

Scenarios

Parse

This functionality is provided by the WinRT API Windows.Foundation.Uri in C++ and JS, and by System.Uri in .NET.
Parsing a URI pulls it apart into its basic components without decoding or otherwise modifying the contents.
var uri = new Windows.Foundation.Uri("http://example.com/path%20segment1/path%20segment2?key1=value1&key2=value2");
console.log(uri.path);// /path%20segment1/path%20segment2

WsDecodeUrl (C++)

WsDecodeUrl is not suitable for general purpose URI parsing.  Use Windows.Foundation.Uri instead.

Build (C#)

URI building is only available in C# via System.UriBuilder.
URI building is the inverse of URI parsing: URI building allows the developer to specify the value of basic components of a URI and the API assembles them into a URI. 
To work around the lack of a URI building API developers will likely concatenate strings to form their URIs.  This can lead to injection bugs if they don’t validate or encode their input properly, but if based on trusted or known input is unlikely to have issues.
            Uri originalUri = new Uri("http://example.com/path1/?query");
            UriBuilder uriBuilder = new UriBuilder(originalUri);
            uriBuilder.Path = "/path2/";
            Uri newUri = uriBuilder.Uri; // http://example.com/path2/?query

WsEncodeUrl (C++)

WsEncodeUrl, in addition to building a URI from components also does some encoding.  It encodes non-US-ASCII characters as UTF8, the percent, and a subset of gen-delims based on the URI property: all :/?#[]@ are percent-encoded except :/@ in the path and :/?@ in query and fragment.
Accordingly, WsEncodeUrl is not suitable for general purpose URI building.  It is acceptable to use in the following cases:
- You’re building a URI out of non-encoded URI properties and don’t care about the difference between encoded and decoded characters.  For instance you’re the only one consuming the URI and you uniformly decode URI properties when consuming – for instance using WsDecodeUrl to consume the URI.
- You’re building a URI with URI properties that don’t contain any of the characters that WsEncodeUrl encodes.

Normalize

This functionality is provided by the WinRT API Windows.Foundation.Uri in C++ and JS and by System.Uri in .NET.  Normalization is applied during construction of the Uri object.
URI normalization is the application of URI normalization rules (including DNS normalization, IDN normalization, percent-encoding normalization, etc.) to the input URI.
        var normalizedUri = new Windows.Foundation.Uri("HTTP://EXAMPLE.COM/p%61th foo/");
        console.log(normalizedUri.absoluteUri); // http://example.com/path%20foo/
This is modulo Win8 812823 in which the Windows.Foundation.Uri.AbsoluteUri property returns a normalized IRI not a normalized URI.  This bug does not affect System.Uri.AbsoluteUri which returns a normalized URI.

Equality

This functionality is provided by the WinRT API Windows.Foundation.Uri in C++ and JS and by System.Uri in .NET. 
URI equality determines if two URIs are equal or not necessarily equal.
            var uri1 = new Windows.Foundation.Uri("HTTP://EXAMPLE.COM/p%61th foo/"),
                uri2 = new Windows.Foundation.Uri("http://example.com/path%20foo/");
            console.log(uri1.equals(uri2)); // true

Relative resolution

This functionality is provided by the WinRT API Windows.Foundation.Uri in C++ and JS and by System.Uri in .NET 
Relative resolution is a function that given an absolute URI A and a relative URI B, produces a new absolute URI C.  C is the combination of A and B in which the basic components specified in B override or combine with those in A under rules specified in RFC 3986.
        var baseUri = new Windows.Foundation.Uri("http://example.com/index.html"),
            relativeUri = "/path?query#fragment",
            absoluteUri = baseUri.combineUri(relativeUri);
        console.log(baseUri.absoluteUri);       // http://example.com/index.html
        console.log(absoluteUri.absoluteUri);   // http://example.com/path?query#fragment

Encode data for including in URI property

This functionality is available in JavaScript via encodeURIComponent and in C# via System.Uri.EscapeDataString. Although the two methods mentioned above will suffice for this purpose, they do not perform exactly the same operation.
Additionally we now have Windows.Foundation.Uri.EscapeComponent in WinRT, which is available in JavaScript and C++ (not C# since it doesn’t have access to Windows.Foundation.Uri).  This is also slightly different from the previously mentioned mechanisms but works best for this purpose.
Encoding data for inclusion in a URI property is necessary when constructing a URI from data.  In all the above cases the developer is dealing with a URI or substrings of a URI and so the strings are all encoded as appropriate. For instance, in the parsing example the path contains “path%20segment1” and not “path segment1”.  To construct a URI one must first construct the basic components of the URI which involves encoding the data.  For example, if one wanted to include “path segment / example” in the path of a URI, one must percent-encode the ‘ ‘ since it is not allowed in a URI, as well as the ‘/’ since although it is allowed, it is a delimiter and won’t be interpreted as data unless encoded.
If a developer does not have this API provided they can write it themselves.  Percent-encoding methods appear simple to write, but the difficult part is getting the set of characters to encode correct, as well as handling non-US-ASCII characters.
        var uri = new Windows.Foundation.Uri("http://example.com" +
            "/" + Windows.Foundation.Uri.escapeComponent("path segment / example") +
            "?key=" + Windows.Foundation.Uri.escapeComponent("=&?#"));
        console.log(uri.absoluteUri); // http://example.com/path%20segment%20%2F%20example?key=%3D%26%3F%23

WsEncodeUrl (C++)

In addition to building a URI from components, WsEncodeUrl also percent-encodes some characters.  However the API is not recommend for this scenario given the particular set of characters that are encoded and the convoluted nature in which a developer would have to use this API in order to use it for this purpose.
There are no general purpose scenarios for which the characters WsEncodeUrl encodes make sense: encode the %, encode a subset of gen-delims but not also encode the sub-delims.  For instance this could not replace encodeURIComponent in a C++ version of the following code snippet since if ‘value’ contained ‘&’ or ‘=’ (both sub-delims) they wouldn’t be encoded and would be confused for delimiters in the name value pairs in the query:
"http://example.com/?key=" + Windows.Foundation.Uri.escapeComponent(value)
Since WsEncodeUrl produces a string URI, to obtain the property they want to encode they’d need to parse the resulting URI.  WsDecodeUrl won’t work because it decodes the property but Windows.Foundation.Uri doesn’t decode.  Accordingly the developer could run their string through WsEncodeUrl then Windows.Foundation.Uri to extract the property.

Decode data extracted from URI property

This functionality is available in JavaScript via decodeURIComponent and in C# via System.Uri.UnescapeDataString. Although the two methods mentioned above will suffice for this purpose, they do not perform exactly the same operation.
Additionally we now also have Windows.Foundation.Uri.UnescapeComponent in WinRT, which is available in JavaScript and C++ (not C# since it doesn’t have access to Windows.Foundation.Uri).  This is also slightly different from the previously mentioned mechanisms but works best for this purpose.
Decoding is necessary when extracting data from a parsed URI property.  For example, if a URI query contains a series of name and value pairs delimited by ‘=’ between names and values, and by ‘&’ between pairs, one must first parse the query into name and value entries and then decode the values.  It is necessary to make this an extra step separate from parsing the URI property so that sub-delimiters (in this case ‘&’ and ‘=’) that are encoded will be interpreted as data, and those that are decoded will be interpreted as delimiters.
If a developer does not have this API provided they can write it themselves.  Percent-decoding methods appear simple to write, but have some tricky parts including correctly handling non-US-ASCII, and remembering not to decode .
In the following example, note that if unescapeComponent were called first, the encoded ‘&’ and ‘=’ would be decoded and interfere with the parsing of the name value pairs in the query.
            var uri = new Windows.Foundation.Uri("http://example.com/?foo=bar&array=%5B%27%E3%84%93%27%2C%27%26%27%2C%27%3D%27%2C%27%23%27%5D");
            uri.query.substr(1).split("&").forEach(
                function (keyValueString) {
                    var keyValue = keyValueString.split("=");
                    console.log(Windows.Foundation.Uri.unescapeComponent(keyValue[0]) + ": " + Windows.Foundation.Uri.unescapeComponent(keyValue[1]));
                    // foo: bar
                    // array: ['','&','=','#']
                });

WsDecodeUrl (C++)

Since WsDecodeUrl decodes all percent-encoded octets it could be used for general purpose percent-decoding but it takes a URI so would require the dev to construct a stub URI around the string they want to decode.  For example they could prefix “http:///#” to their string, run it through WsDecodeUrl and then extract the fragment property.  It is convoluted but will work correctly.

Parse Query

The query of a URI is often encoded as application/x-www-form-urlencoded which is percent-encoded name value pairs delimited by ‘&’ between pairs and ‘=’ between corresponding names and values.
In WinRT we have a class to parse this form of encoding using Windows.Foundation.WwwFormUrlDecoder.  The queryParsed property on the Windows.Foundation.Uri class is of this type and created with the query of its Uri:
    var uri = Windows.Foundation.Uri("http://example.com/?foo=bar&array=%5B%27%E3%84%93%27%2C%27%26%27%2C%27%3D%27%2C%27%23%27%5D");
    uri.queryParsed.forEach(
        function (pair) {
            console.log("name: " + pair.name + ", value: " + pair.value);
            // name: foo, value: bar
            // name: array, value: ['','&','=','#']
        });
    console.log(uri.queryParsed.getFirstValueByName("array")); // ['','&','=','#']
The QueryParsed property is only on Windows.Foundation.Uri and not System.Uri and accordingly is not available in .NET.  However the Windows.Foundation.WwwFormUrlDecoder class is available in C# and can be used manually:
            Uri uri = new Uri("http://example.com/?foo=bar&array=%5B%27%E3%84%93%27%2C%27%26%27%2C%27%3D%27%2C%27%23%27%5D");
            WwwFormUrlDecoder decoder = new WwwFormUrlDecoder(uri.Query);
            foreach (IWwwFormUrlDecoderEntry entry in decoder)
            {
                System.Diagnostics.Debug.WriteLine("name: " + entry.Name + ", value: " + entry.Value);
                // name: foo, value: bar
                // name: array, value: ['','&','=','#']
            }
 

Build Query

To build a query of name value pairs encoded as application/x-www-form-urlencoded there is no WinRT API to do this directly.  Instead a developer must do this manually making use of the code described in “Encode data for including in URI property”.
In terms of public releases, this property is only in the RC and later builds.
For example in JavaScript a developer may write:
            var uri = new Windows.Foundation.Uri("http://example.com/"),
                query = "?" + Windows.Foundation.Uri.escapeComponent("array") + "=" + Windows.Foundation.Uri.escapeComponent("['','&','=','#']");
 
            console.log(uri.combine(new Windows.Foundation.Uri(query)).absoluteUri); // http://example.com/?array=%5B'%E3%84%93'%2C'%26'%2C'%3D'%2C'%23'%5D
 
PermalinkCommentsc# c++ javascript technical uri windows windows-runtime windows-store

Subtleties of postMessage

2013 Jul 15, 1:00

In IE10 and other new browsers one may create MessageChannel objects that have two MessagePorts each connected (w3c spec calls it entangled) to one another such that postMessage on one port results in the message event firing on the other. You can pass an array of ports as the last parameter to postMessage and they show up in the ports property of the message event arg.

Origin

The postMessage here is like the worker postMessage and unlike the window and iframe postMessage in that it applies no origin checking:

  1. No origin postMessage in workers and MessagePorts: postMessage(messageData, ports)
  2. Origin postMessage in windows and iframes: postMessage(messageData, targetOrigin, ports)

Unfortunately the origin isn't an optional parameter at the end to make the two postMessages have the same signature.

On the event handler side, the event arg always has an origin property. But in the no origin case it is always the empty string.

Source

There is also a source property on the message event arg which if set is an object that has a postMessage property allowing you to post back to your caller. It is set for the origin case, however, in the no origin case this property is null. This is somewhat reasonable because in the case of MessagePort and Workers there are only two endpoints so you always know the source of a message implicitly. Unlike the origin case in which any iframe or window can be calling postMessage on any other iframe or window and the caller is unknown. So not unreasonable but it would be nice if the source property was always set for consistency.

MessageChannel start

When a MessageChannel is created it has two MessagePorts, but until those ports are started they will queue up any messages they receive. Once started they will dispatch all queued messages. Ports don't have to be started to send messages.

A port may be started in two ways, either by explicitly calling the start method on the port, or by setting the onmessage callback property on the port. However, adding an event listener via addEventListener("message", does not start the port. It works this way in IE and Chrome and the spec states this as well.

The justification is that since you can have only one callback via onmessage that once set you must implicitly be ready to receive messages and its fine to start the port. As opposed to the addEventListener in which case the user agent cannot start implicitly because it doesn't know how many event listeners will be added.  I found Hixie stating this justification in geoloc meeting notes.

Links

W3C Spec

Opera introduction

PermalinkCommentsDOM html javascript postMessage technical web-worker worker

WinRT PropertySet Changed Event Danger

2013 Jul 8, 1:46

The Windows Runtime API Windows.Foundation.Collections.PropertySet class​ is a nice string name to object value map that has a changed event that fires when the contents of the map is modified. Be careful with this event because it fires synchronously from the thread on which the PropertySet was modified. If modified from the UI thread, the UI thread will then wait as it synchronously dispatches the changed event to all listeners which could lead to performance issues or especially from the UI thread deadlock. For instance, deadlock if you have two threads both trying to tell each other about changed events for different PropertySets.

PermalinkCommentsdeadlock development propertyset windows windows-runtime winrt

Words with Hints Windows 8 App Development Notes

2013 Jul 4, 1:00

My second completed app for the Windows Store was Words with Hints a companion to Words with Friends or other Scrabble like games that gives you *ahem* hints. You provide your tiles and optionally letters placed in a line on the board and Words with Hints gives you word options.

I wrote this the first time by building a regular expression to check against my dictionary of words which made for a slow app on the Surface. In subsequent release of the app I now spawn four web workers (one for each of the Surface's cores) each with its own fourth of my dictionary. Each fourth of the dictionary is a trie which makes it easy for me to discard whole chunks of possible combinations of Scrabble letters as I walk the tree of possibilities.

The dictionaries are large and takes a noticeable amount of time to load on the Surface. The best performing mechanism I found to load them is as JavaScript source files that simply define their portion of the dictionary on the global object and synchronously (only on the worker so not blocking the UI thread). Putting them into .js files means they take advantage of bytecode caching making them load faster. However because the data is mostly strings and not code there is a dramatic size increase when the app is installed. The total size of the four dictionary .js files is about 44Mb. The bytecode cache for the dictionary files is about double that 88Mb meaning the dictionary plus the bytecode cache is 132Mb.

To handle the bother of postMessage communication and web workers this was the first app in which I used my promise MessagePort project which I'll discuss more in the future.

This is the first app in which I used the Microsoft Ad SDK. It was difficult to find the install for the SDK and difficult to use their website, but once setup, the Ad SDK was easy to import into VS and easy to use in my app.

PermalinkCommentsdevelopment technical windows windows-store words-with-hints

Shout Text Windows 8 App Development Notes

2013 Jun 27, 1:00

My first app for Windows 8 was Shout Text. You type into Shout Text, and your text is scaled up as large as possible while still fitting on the screen, as you type. It is the closest thing to a Hello World app as you'll find on the Windows Store that doesn't contain that phrase (by default) and I approached it as the simplest app I could make to learn about Windows modern app development and Windows Store app submission.

I rely on WinJS's default layout to use CSS transforms to scale up the user's text as they type. And they are typing into a simple content editable div.

The app was too simple for me to even consider using ads or charging for it which I learned more about in future apps.

The first interesting issue I ran into was that copying from and then pasting into the content editable div resulted in duplicates of the containing div with copied CSS appearing recursively inside of the content editable div. To fix this I had to catch the paste operation and remove the HTML data from the clipboard to ensure only the plain text data is pasted:

        function onPaste() {
var text;

if (window.clipboardData) {
text = window.clipboardData.getData("Text").toString();
window.clipboardData.clearData("Html");
window.clipboardData.setData("Text", util.normalizeContentEditableText(text));
}
}
shoutText.addEventListener("beforepaste", function () { return false; }, false);
shoutText.addEventListener("paste", onPaste, false);

I additionally found an issue in IE in which applying a CSS transform to a content editable div that has focus doesn't move the screen position of the user input caret - the text is scaled up or down but the caret remains the same size and in the same place on the screen. To fix this I made the following hack to reapply the current cursor position and text selection which resets the screen position of the user input caret.

        function resetCaret() {
setTimeout(function () {
var cursorPos = document.selection.createRange().duplicate();
cursorPos.select();
}, 200);
}

shoutText.attachEvent("onresize", function () { resetCaret(); }, true);
PermalinkCommentsdevelopment html javascript shout-text technical windows windows-store

Stripe CTF - XSS, CSRF (Levels 4 & 6)

2012 Sep 10, 4:43

Level 4 and level 6 of the Stripe CTF had solutions around XSS.

Level 4

Code

> Registered Users 

    <%@registered_users.each do |user| %>
    <%last_active = user[:last_active].strftime('%H:%M:%S UTC') %>
    <%if @trusts_me.include?(user[:username]) %>

  • <%= user[:username] %>
    (password: <%= user[:password] %>, last active <%= last_active %>)
  • Issue

    The level 4 web application lets you transfer karma to another user and in doing so you are also forced to expose your password to that user. The main user page displays a list of users who have transfered karma to you along with their password. The password is not HTML encoded so we can inject HTML into that user's browser. For instance, we could create an account with the following HTML as the password which will result in XSS with that HTML:

    
    
    This HTML runs script that uses jQuery to post to the transfer URI resulting in a transfer of karma from the attacked user to the attacker user, and also the attacked user's password.

    Notes

    Code review red flags in this case included lack of encoding when using user controlled content to create HTML content, storing passwords in plain text in the database, and displaying passwords generally. By design the web app shows users passwords which is a very bad idea.

    Level 6

    Code

    
    

    ...

    def self.safe_insert(table, key_values)
    key_values.each do |key, value|
    # Just in case people try to exfiltrate
    # level07-password-holder's password
    if value.kind_of?(String) &&
    (value.include?('"') || value.include?("'"))
    raise "Value has unsafe characters"
    end
    end

    conn[table].insert(key_values)
    end

    Issue

    This web app does a much better job than the level 4 app with HTML injection. They use encoding whenever creating HTML using user controlled data, however they don't use encoding when injecting JSON data into script (see post_data initialization above). This JSON data is the last five most recent messages sent on the app so we get to inject script directly. However, the system also ensures that no strings we write contains single or double quotes so we can't get out of the string in the JSON data directly. As it turns out, HTML lets you jump out of a script block using no matter where you are in script. For instance, in the middle of a value in some JSON data we can jump out of script. But we still want to run script, so we can jump right back in. So the frame so far for the message we're going to post is the following:

    
    
    
    
PermalinkCommentscsrf encoding html internet javascript percent-encoding script security stripe-ctf technical web xss

Stripe CTF - Input validation (Levels 1 & 2)

2012 Sep 6, 5:00

Stripe's web security CTF's Level 1 and level 2 of the Stripe CTF had issues with missing input validation solutions described below.

Level 1

Code

          $filename = 'secret-combination.txt';
extract($_GET);
if (isset($attempt)) {
$combination = trim(file_get_contents($filename));
if ($attempt === $combination) {

Issue

The issue here is the usage of the extract php method which extracts name value pairs from the map input parameter and creates corresponding local variables. However this code uses $_GET which contains a map of name value pairs passed in the query of the URI. The expected behavior is to get an attempt variable out, but since no input validation is done I can provide a filename variable and overwrite the value of $filename. Providing an empty string gives an empty string $combination which I can match with an empty string $attempt. So without knowing the combination I can get past the combination check.

Notes

Code review red flag in this case was the direct use of $_GET with no validation. Instead of using extract the developer could try to extract specifically the attempt variable manually without using extract.

Level 2

Code

    $dest_dir = "uploads/";
$dest = $dest_dir . basename($_FILES["dispic"]["name"]);
$src = $_FILES["dispic"]["tmp_name"];
if (move_uploaded_file($src, $dest)) {
$_SESSION["dispic_url"] = $dest;
chmod($dest, 0644);
echo "

Successfully uploaded your display picture.

";
}

Issue

This code accepts POST uploads of images but with no validation to ensure it is not an arbitrary file. And even though it uses chmod to ensure the file is not executable, things like PHP don't require a file to be executable in order to run them. Accordingly, one can upload a PHP script, then navigate to that script to run it. My PHP script dumped out the contents of the file we're interested in for this level:

Notes

Code review red flags include manual file management, chmod, and use of file and filename inputs without any kind of validation. If this code controlled the filename and ensured that the extension was one of a set of image extensions, this would solve this issue. Due to browser mime sniffing its additionally a good idea to serve a content-type that starts with "image/" for these uploads to ensure browsers treat these as images and not sniff for script or HTML.

PermalinkCommentsinput-validation php security technical

Stripe CTF - SQL injections (Levels 0 & 3)

2012 Sep 5, 9:10

Stripe's web security CTF's level 0 and level 3 had SQL injection solutions described below.

Level 0

Code

app.get('/*', function(req, res) {
var namespace = req.param('namespace');

if (namespace) {
var query = 'SELECT * FROM secrets WHERE key LIKE ? || ".%"';
db.all(query, namespace, function(err, secrets) {

Issue

There's no input validation on the namespace parameter and it is injected into the SQL query with no encoding applied. This means you can use the '%' character as the namespace which is the wildcard character matching all secrets.

Notes

Code review red flag was using strings to query the database. Additional levels made this harder to exploit by using an API with objects to construct a query rather than strings and by running a query that only returned a single row, only ran a single command, and didn't just dump out the results of the query to the caller.

Level 3

Code

@app.route('/login', methods=['POST'])
def login():
username = flask.request.form.get('username')
password = flask.request.form.get('password')

if not username:
return "Must provide username\n"

if not password:
return "Must provide password\n"

conn = sqlite3.connect(os.path.join(data_dir, 'users.db'))
cursor = conn.cursor()

query = """SELECT id, password_hash, salt FROM users
WHERE username = '{0}' LIMIT 1""".format(username)
cursor.execute(query)

res = cursor.fetchone()
if not res:
return "There's no such user {0}!\n".format(username)
user_id, password_hash, salt = res

calculated_hash = hashlib.sha256(password + salt)
if calculated_hash.hexdigest() != password_hash:
return "That's not the password for {0}!\n".format(username)

Issue

There's little input validation on username before it is used to constrcut a SQL query. There's no encoding applied when constructing the SQL query string which is used to, given a username, produce the hashed password and the associated salt. Accordingly one can make username a part of a SQL query command which ensures the original select returns nothing and provide a new SELECT via a UNION that returns some literal values for the hash and salt. For instance the following in blue is the query template and the red is the username injected SQL code:

SELECT id, password_hash, salt FROM users WHERE username = 'doesntexist' UNION SELECT id, ('5e884898da28047151d0e56f8dc6292773603d0d6aabbdd62a11ef721d1542d8') AS password_hash, ('word') AS salt FROM users WHERE username = 'bob' LIMIT 1
In the above I've supplied my own salt and hash such that my salt (word) plus my password (pass) hashed produce the hash I provided above. Accordingly, by providing the above long and interesting looking username and password as 'pass' I can login as any user.

Notes

Code review red flag is again using strings to query the database. Although this level was made more difficult by using an API that returns only a single row and by using the execute method which only runs one command. I was forced to (as a SQL noob) learn the syntax of SELECT in order to figure out UNION and how to return my own literal values.

PermalinkCommentssecurity sql sql-injection technical web-security

Stripe Web Security CTF Summary

2012 Aug 30, 5:00

I was the 546th person to complete Stripe's web security CTF and again had a ton of fun applying my theoretical knowledge of web security issues to the (semi-)real world. As I went through the levels I thought about what red flags jumped out at me (or should have) that I could apply to future code reviews:

Level Issue Code Review Red Flags
0 Simple SQL injection No encoding when constructing SQL command strings. Constructing SQL command strings instead of SQL API
1 extract($_GET); No input validation.
2 Arbitrary PHP execution No input validation. Allow file uploads. File permissions modification.
3 Advanced SQL injection Constructing SQL command strings instead of SQL API.
4 HTML injection, XSS and CSRF No encoding when constructing HTML. No CSRF counter measures. Passwords stored in plain text. Password displayed on site.
5 Pingback server doesn't need to opt-in n/a - By design protocol issue.
6 Script injection and XSS No encoding while constructing script. Deny list (of dangerous characters). Passwords stored in plain text. Password displayed on site.
7 Length extension attack Custom crypto code. Constructing SQL command string instead of SQL API.
8 Side channel attack Password handling code. Timing attack mitigation too clever.

More about each level in the future.

PermalinkCommentscode-review coding csrf html internet programming script security sql stripe technical web xss
Older Entries Creative Commons License Some rights reserved.