Changing the User Agent string in UWP WebView

2018 Oct 23, 9:32

There's no perfect way to change the user agent string for the UWP WebView (x-ms-webview in HTML, Windows.UI.Xaml.Controls.WebView in XAML, and Windows.Web.UI.Interop.WebViewControl in Win32) but there are two imperfect methods folks end up using.

The first is to call UrlMkSetSessionOption. This is an old public API that allows you to configure various arcane options including one that is the default user agent string for requests running through urlmon. This API is allowed by the Microsoft Store for UWP apps. The change it applies is process wide which has two potential drawbacks. If you want to be able to have different UA strings set for different requests from a WebView that's not really possible with this solution. The other drawback is if you're using out of process WebView, you need to ensure you're calling into UrlMkSetSessionOption in the WebView's process. You'll need to write third party WinRT that calls UrlMkSetSessionOption, create the out of proc WebView, navigate it to some trusted local page, use AddWebAllowedObject or provide that URI WinRT access, and call into your third party WinRT. You'll need to do that for any new WebView process you create.

The second less generally applicable solution is to use NavigateWithHttpRequestMessage and set the User-Agent HTTP header. In this case you get to control the scope of the user agent string changes but has the limitations that not all sub resource downloads will use this user agent string and for navigations you don't initiate you have to manually intercept and re-request being careful to transfer over all POST body state and HTTP headers correctly. That last part is not actually possible for iframes.

Tweet from David_Risney

2016 Jan 27, 9:57

FCC proposes replacing cable card with software based equivalent. Would be amazing if it could happen. http://arstechnica.com/business/2016/01/cable-lobby-is-really-mad-about-fccs-set-top-box-competition-plan/ …

Tweet from David_Risney

2015 Aug 4, 3:06

Enjoying this inspirational cord cutting Ars article: http://arstechnica.com/staff/2015/08/i-finally-kicked-cable-tv-to-the-curb/ …. Can I use HBO Now subscription with HBO Go app on Xbox One?

Retweet of TheOnion

2015 Mar 9, 12:11

How Cable Companies Plan To Fight Cord Cutting http://onion.com/1aXT9RD pic.twitter.com/1e2qYRFCMN

URI functions in Windows Store Applications

2013 Jul 25, 1:00

Summary

The Modern SDK contains some URI related functionality as do libraries available in particular projection languages. Unfortunately, collectively these APIs do not cover all scenarios in all languages. Specifically, JavaScript and C++ have no URI building APIs, and C++ additionally has no percent-encoding/decoding APIs.

	WinRT (JS and C++)	JS Only	C++ Only	.NET Only
Parse	Windows.Foundation.Uri		WsDecodeUrl*	System.Uri
Build			WsEncodeUrl*	System.UriBuilder
Normalize	Windows.Foundation.Uri			System.Uri
Equality	Windows.Foundation.Uri. Equals			System.Uri.Compare
Relative resolution	Windows.Foundation.Uri. CombineUri			System.Uri
Encode data for including in URI property	Windows.Foundation.Uri. EscapeComponent	encodeURIComponent	WsEncodeUrl*	System.Uri. EscapeDataString
Decode data extracted from URI property	Windows.Foundation.Uri. UnescapeComponent	decodeURIComponent	WsDecodeUrl*	System.Uri. UnescapeDataString
Build Query
Parse Query	Windows.Foundation. Uri.QueryParsed / Windows.Foundation. WwwFormUrlDecoder*

The Windows.Foudnation.Uri type is not projected into .NET modern applications. Instead those applications use System.Uri and the platform ensures that it is correctly converted back and forth between Windows.Foundation.Uri as appropriate. Accordingly the column marked WinRT above is applicable to JS and C++ modern applications but not .NET modern applications. The only entries above applicable to .NET are the .NET Only column and the WwwFormUrlDecoder in the bottom left which is available to .NET.

Scenarios

Parse

This functionality is provided by the WinRT API Windows.Foundation.Uri in C++ and JS, and by System.Uri in .NET.

Parsing a URI pulls it apart into its basic components without decoding or otherwise modifying the contents.

var uri = new Windows.Foundation.Uri("http://example.com/path%20segment1/path%20segment2?key1=value1&key2=value2");

console.log(uri.path);// /path%20segment1/path%20segment2

WsDecodeUrl (C++)

WsDecodeUrl is not suitable for general purpose URI parsing. Use Windows.Foundation.Uri instead.

Build (C#)

URI building is only available in C# via System.UriBuilder.

URI building is the inverse of URI parsing: URI building allows the developer to specify the value of basic components of a URI and the API assembles them into a URI.

To work around the lack of a URI building API developers will likely concatenate strings to form their URIs. This can lead to injection bugs if they don’t validate or encode their input properly, but if based on trusted or known input is unlikely to have issues.

Uri originalUri = new Uri("http://example.com/path1/?query");

UriBuilder uriBuilder = new UriBuilder(originalUri);

uriBuilder.Path = "/path2/";

Uri newUri = uriBuilder.Uri; // http://example.com/path2/?query

WsEncodeUrl (C++)

WsEncodeUrl, in addition to building a URI from components also does some encoding. It encodes non-US-ASCII characters as UTF8, the percent, and a subset of gen-delims based on the URI property: all :/?#[]@ are percent-encoded except :/@ in the path and :/?@ in query and fragment.

Accordingly, WsEncodeUrl is not suitable for general purpose URI building. It is acceptable to use in the following cases:

- You’re building a URI out of non-encoded URI properties and don’t care about the difference between encoded and decoded characters. For instance you’re the only one consuming the URI and you uniformly decode URI properties when consuming – for instance using WsDecodeUrl to consume the URI.

- You’re building a URI with URI properties that don’t contain any of the characters that WsEncodeUrl encodes.

Normalize

This functionality is provided by the WinRT API Windows.Foundation.Uri in C++ and JS and by System.Uri in .NET. Normalization is applied during construction of the Uri object.

URI normalization is the application of URI normalization rules (including DNS normalization, IDN normalization, percent-encoding normalization, etc.) to the input URI.

var normalizedUri = new Windows.Foundation.Uri("HTTP://EXAMPLE.COM/p%61th foo/");

console.log(normalizedUri.absoluteUri); // http://example.com/path%20foo/

This is modulo Win8 812823 in which the Windows.Foundation.Uri.AbsoluteUri property returns a normalized IRI not a normalized URI. This bug does not affect System.Uri.AbsoluteUri which returns a normalized URI.

Equality

This functionality is provided by the WinRT API Windows.Foundation.Uri in C++ and JS and by System.Uri in .NET.

URI equality determines if two URIs are equal or not necessarily equal.

var uri1 = new Windows.Foundation.Uri("HTTP://EXAMPLE.COM/p%61th foo/"),

uri2 = new Windows.Foundation.Uri("http://example.com/path%20foo/");

console.log(uri1.equals(uri2)); // true

Relative resolution

This functionality is provided by the WinRT API Windows.Foundation.Uri in C++ and JS and by System.Uri in .NET

Relative resolution is a function that given an absolute URI A and a relative URI B, produces a new absolute URI C. C is the combination of A and B in which the basic components specified in B override or combine with those in A under rules specified in RFC 3986.

var baseUri = new Windows.Foundation.Uri("http://example.com/index.html"),

relativeUri = "/path?query#fragment",

absoluteUri = baseUri.combineUri(relativeUri);

console.log(baseUri.absoluteUri); // http://example.com/index.html

console.log(absoluteUri.absoluteUri); // http://example.com/path?query#fragment

Encode data for including in URI property

This functionality is available in JavaScript via encodeURIComponent and in C# via System.Uri.EscapeDataString. Although the two methods mentioned above will suffice for this purpose, they do not perform exactly the same operation.

Additionally we now have Windows.Foundation.Uri.EscapeComponent in WinRT, which is available in JavaScript and C++ (not C# since it doesn’t have access to Windows.Foundation.Uri). This is also slightly different from the previously mentioned mechanisms but works best for this purpose.

Encoding data for inclusion in a URI property is necessary when constructing a URI from data. In all the above cases the developer is dealing with a URI or substrings of a URI and so the strings are all encoded as appropriate. For instance, in the parsing example the path contains “path%20segment1” and not “path segment1”. To construct a URI one must first construct the basic components of the URI which involves encoding the data. For example, if one wanted to include “path segment / example” in the path of a URI, one must percent-encode the ‘ ‘ since it is not allowed in a URI, as well as the ‘/’ since although it is allowed, it is a delimiter and won’t be interpreted as data unless encoded.

If a developer does not have this API provided they can write it themselves. Percent-encoding methods appear simple to write, but the difficult part is getting the set of characters to encode correct, as well as handling non-US-ASCII characters.

var uri = new Windows.Foundation.Uri("http://example.com" +

"/" + Windows.Foundation.Uri.escapeComponent("path segment / example") +

"?key=" + Windows.Foundation.Uri.escapeComponent("=&?#"));

console.log(uri.absoluteUri); // http://example.com/path%20segment%20%2F%20example?key=%3D%26%3F%23

WsEncodeUrl (C++)

In addition to building a URI from components, WsEncodeUrl also percent-encodes some characters. However the API is not recommend for this scenario given the particular set of characters that are encoded and the convoluted nature in which a developer would have to use this API in order to use it for this purpose.

There are no general purpose scenarios for which the characters WsEncodeUrl encodes make sense: encode the %, encode a subset of gen-delims but not also encode the sub-delims. For instance this could not replace encodeURIComponent in a C++ version of the following code snippet since if ‘value’ contained ‘&’ or ‘=’ (both sub-delims) they wouldn’t be encoded and would be confused for delimiters in the name value pairs in the query:

"http://example.com/?key=" + Windows.Foundation.Uri.escapeComponent(value)

Since WsEncodeUrl produces a string URI, to obtain the property they want to encode they’d need to parse the resulting URI. WsDecodeUrl won’t work because it decodes the property but Windows.Foundation.Uri doesn’t decode. Accordingly the developer could run their string through WsEncodeUrl then Windows.Foundation.Uri to extract the property.

Decode data extracted from URI property

This functionality is available in JavaScript via decodeURIComponent and in C# via System.Uri.UnescapeDataString. Although the two methods mentioned above will suffice for this purpose, they do not perform exactly the same operation.

Additionally we now also have Windows.Foundation.Uri.UnescapeComponent in WinRT, which is available in JavaScript and C++ (not C# since it doesn’t have access to Windows.Foundation.Uri). This is also slightly different from the previously mentioned mechanisms but works best for this purpose.

Decoding is necessary when extracting data from a parsed URI property. For example, if a URI query contains a series of name and value pairs delimited by ‘=’ between names and values, and by ‘&’ between pairs, one must first parse the query into name and value entries and then decode the values. It is necessary to make this an extra step separate from parsing the URI property so that sub-delimiters (in this case ‘&’ and ‘=’) that are encoded will be interpreted as data, and those that are decoded will be interpreted as delimiters.

If a developer does not have this API provided they can write it themselves. Percent-decoding methods appear simple to write, but have some tricky parts including correctly handling non-US-ASCII, and remembering not to decode .

In the following example, note that if unescapeComponent were called first, the encoded ‘&’ and ‘=’ would be decoded and interfere with the parsing of the name value pairs in the query.

var uri = new Windows.Foundation.Uri("http://example.com/?foo=bar&array=%5B%27%E3%84%93%27%2C%27%26%27%2C%27%3D%27%2C%27%23%27%5D");

uri.query.substr(1).split("&").forEach(

function (keyValueString) {

var keyValue = keyValueString.split("=");

console.log(Windows.Foundation.Uri.unescapeComponent(keyValue[0]) + ": " + Windows.Foundation.Uri.unescapeComponent(keyValue[1]));

// foo: bar

// array: ['ㄓ','&','=','#']

});

WsDecodeUrl (C++)

Since WsDecodeUrl decodes all percent-encoded octets it could be used for general purpose percent-decoding but it takes a URI so would require the dev to construct a stub URI around the string they want to decode. For example they could prefix “http:///#” to their string, run it through WsDecodeUrl and then extract the fragment property. It is convoluted but will work correctly.

Parse Query

The query of a URI is often encoded as application/x-www-form-urlencoded which is percent-encoded name value pairs delimited by ‘&’ between pairs and ‘=’ between corresponding names and values.

In WinRT we have a class to parse this form of encoding using Windows.Foundation.WwwFormUrlDecoder. The queryParsed property on the Windows.Foundation.Uri class is of this type and created with the query of its Uri:

var uri = Windows.Foundation.Uri("http://example.com/?foo=bar&array=%5B%27%E3%84%93%27%2C%27%26%27%2C%27%3D%27%2C%27%23%27%5D");

uri.queryParsed.forEach(

function (pair) {

console.log("name: " + pair.name + ", value: " + pair.value);

// name: foo, value: bar

// name: array, value: ['ㄓ','&','=','#']

});

console.log(uri.queryParsed.getFirstValueByName("array")); // ['ㄓ','&','=','#']

The QueryParsed property is only on Windows.Foundation.Uri and not System.Uri and accordingly is not available in .NET. However the Windows.Foundation.WwwFormUrlDecoder class is available in C# and can be used manually:

Uri uri = new Uri("http://example.com/?foo=bar&array=%5B%27%E3%84%93%27%2C%27%26%27%2C%27%3D%27%2C%27%23%27%5D");

WwwFormUrlDecoder decoder = new WwwFormUrlDecoder(uri.Query);

foreach (IWwwFormUrlDecoderEntry entry in decoder)

{

System.Diagnostics.Debug.WriteLine("name: " + entry.Name + ", value: " + entry.Value);

// name: foo, value: bar

// name: array, value: ['ㄓ','&','=','#']

}

Build Query

To build a query of name value pairs encoded as application/x-www-form-urlencoded there is no WinRT API to do this directly. Instead a developer must do this manually making use of the code described in “Encode data for including in URI property”.

In terms of public releases, this property is only in the RC and later builds.

For example in JavaScript a developer may write:

var uri = new Windows.Foundation.Uri("http://example.com/"),

query = "?" + Windows.Foundation.Uri.escapeComponent("array") + "=" + Windows.Foundation.Uri.escapeComponent("['ㄓ','&','=','#']");

console.log(uri.combine(new Windows.Foundation.Uri(query)).absoluteUri); // http://example.com/?array=%5B'%E3%84%93'%2C'%26'%2C'%3D'%2C'%23'%5D

c# c++ javascript technical uri windows windows-runtime windows-store

I'm an American and I want to watch the Olympics. What do I do? (iamnotaprogrammer.com)

2012 Jul 28, 12:05

One persons quest to watch the Olympics online.

The location requirements (guessed at via IP address) are irritating. The requirement that you have a particular cable subscription to view video online seems like not network neutrality.

Also this related article:

http://techcrunch.com/2012/07/27/nbc-olympic-opening-ceremony/

olympics video internet web

Matter Anti-Matter: The Revolution Will Not Be Televised

2012 Jun 7, 3:07

ensignau:

The conversation surrounding how to liberate HBO from its cable partners and create the entertainment utopia viewers have long desired has been fascinating.

The resulting analyses of the numbers has pretty much shown that the amount people are willing to pay is not nearly enough to…

tv economics youtube video internet web

draft-liman-tld-names-06 - Top Level Domain Name Specification

2011 Dec 4, 3:00

“The syntax for allowed Top-Level Domain (TLD) labels in the Domain Name System (DNS) is not clearly applicable to the encoding of Internationalised Domain Names (IDNs) as TLDs. This document provides a concise specification of TLD label syntax based on existing syntax documentation, extended minimally to accommodate IDNs.” Still irritated about arbitrary TLDs.

technical syntax dns tld idn

Experiences from an IPv6-Only Network

2011 Apr 30, 4:05"This document discusses our experiences from moving a small number of users to an IPv6-only network, with access to the IPv4-only parts of the Internet via a NAT64 device. The document covers practical experiences as well as road blocks and opportunities for this type of a network setup. The document also makes some recommendations about where such networks are applicable and what should be taken into account in the network design. The document also discusses further work that is needed to make IPv6-only networking applicable in all environments."

internet ip ipv6 ipv4 nat technical reference

Multi-Stream Tuner Cards by Ceton

2010 Dec 8, 5:43This that lets your PC read four digital encrypted cable signals at once with a cable card from you cable company. This is cool although expensive. What frustrates me is the lack of choices in this area.

cable cablecard hardware pc technical mediacenter

Methods of Watching TV

2010 Feb 3, 1:09If you've got cable, Internet, and Netflix you end up with a large number of TV viewing options. Its nice to have the options but is there some way to collect and summarize my available options at any one time?

	TV	PC
Comcast TV	Xbox 360 + Windows Media Center	Windows Media Center
Comcast OnDemand	Cable box	Fancast
Hulu	Xbox 360 + PlayOn + Windows Media Center	Hulu
Netflix Watch Instantly	Xbox 360 + Netflix App	Netflix Watch Instantly
Netflix DVDs	Xbox 360	PC

tv xbox 360 wmc dvr

Joho the Blog » Broadband. Trust them.

2009 Sep 25, 5:18"The closest the organization comes to stating its actual intent is in the wording of the print ad they’re running. Hmm. On the open medium of the Internet the organization hides its purpose, but in the controlled medium of print, they come close to stating it. How unexpected!"

net-neutrality network-neutrality network internet broadband isp cable humor

Comcast Digital Switch Impact on My Windows Media Center

2009 Sep 25, 2:19

Irritatingly out of line with what their commercials say, in my area Comcast, under the covers of the national broadcast digital switch, is sneaking in their own switch to digital, moving channels above 30 to their own digital format. Previously, I had Windows 7 Media Center running on a PC with a Hauppauge PVR500 which can decode two television signals at once setup to record shows I like. The XBox 360 works great as a Media Center client letting me easily watch the recorded shows over my home network on my normal TV.

Unfortunately with Comcast's change, now one needs a cable box or a Comcast digital to analog converter in order to view their signal, but Comcast is offering up to two free converters for those who'd like them. The second of my two free converters I hooked up to the Media Center PC and I got the IR Blaster that came with my Hauppauge out of the garage. I plugged in the USB IR Blaster to my PC, connected one of the IR transmitters to the 1st port on the IR Blaster, and sat the IR transmitter next to the converter's IR receiver. I went through the Media Center TV setup again and happily it was able to figure out how to correctly change the channel on the converter. So I can record now, however:

I can only record one thing at a time now
Changing the channel is slow taking many seconds (no flipping through channels for me)
The Hauppauge card can't know if the channel change worked. So if it tries to change to HBO (I get it for free with one of the Comcast packages) which is encrypted and the converted won't show, the channel doesn't change but the PC doesn't know it and ends up recording some other channel.

To fix (3) I need to manually go through and remove channels I don't have from the Media Center. To fix (1) I may be able to get a second IR transmitter, a third digital converter, hook it up to one of the other inputs on my Hauppauge, and go back through the Media Center TV setup. There's no fix for (2) but that's not so bad. All in all, its just generally frustrating that they're breaking my setup with no obvious benefit.

digital tv hauppauge mce cable windows media center comcast

Brier Dudley's blog | More on Comcast digital: TiVos, Media Centers, HD ...

2009 Sep 20, 10:44Grumble grumble...

comcast digital cable windows mediacenter tv television tivo

CableCARD now a go for homebrew home theater PCs - Ars Technica

2009 Sep 10, 5:52Just in time for Comcast switching channels above 30 to digital only. My current Windows Media Center setup will not function soon. How much do you think I'm going to have to spend to get it working now...

cablecard cable tv windows mediacenter media technical

Mostly Moved Into New House

2009 Jun 19, 8:07

The weekend before the previous, Sarah and I moved our belongings into the new house and spent a lot of time packing and unpacking, and now we're officially living there (interested Facebook friends can find my new address or just ask me). The Saturday of the previous weekend Sarah's family came over for a half house warming and half Sarah's birthday celebration which was fun and served to force us to do more unpacking and forced me to take trips to Home Depot, Bed Bath and Beyond, etc. On Sunday, Sarah and I went out to her favorite restaurant and she opened her gifts that I had to hide to keep her from opening before her birthday. Happy Birthday Sarah!

While at Home Depot I had trouble finding what I was actually looking for, but I did find everything I needed to terminate the Cat5e cables that are wired in the house. Each room has a wall plate with two RJ45 sockets, both sockets wired to Cat5e cable. One of the cables per plate was already hooked up to a standard phone service punchdown board and the other cables per plate were all hanging unterminated next to the punchdown board. So now I've terminated them all with RJ45 connectors and hooked them up to my hub, wireless router, cable modem, etc. I had the same sort of fun setting all that up as I did playing with model train sets as a child. Hopefully no therapy will be required to figure out why that is.

personal2 train address sarah house new-house birthday

A Brief History of Microsoft's Live Search's New Domain Bing

2009 Jun 1, 11:07 Logo for bing! from 2003 via The Wayback Machine

When I heard that Live Search is now Bing one of my initial thoughts was how'd they get that domain name given the unavailability of pronouncable four letter .COM domain names. Well, the names been used in the past. Here now, via the Wayback Machine is a brief, somewhat speculative, and ultimately anticlimactic history of bing.com:

2003 July: The first archived version of bing.com features "bing! is a small device (e.g., possibly even a small Band-Aid(R)-like sticker!) that vibrates when a person's cell phone rings." I can't recall 2003 cell phones, were they big enough to require this device?
2004 August: Site for the same device is rewritten and looks much better, IMHO.
2006 June: The domain is now parked by easyDNS. I guess the "bing!" device didn't work out?
2006 November: Its now "BING*" and they won't say what they're working on ("we're still in stealth mode") but they are hiring C#/.NET developers.
2007 January: And they're gone. Without even exiting stealth mode. Too bad, I liked their logo. Their domain is now for sale...
2007 February: Looks like EasyMail buys the domain and offers a physical mailing service in Australia: "By simply clicking a button on your computer, mail is beamed electronically to a bing post office. Your mail is automatically printed, folded, enveloped and dispatched into the Australia Post network the very same day."
Present: Now its the new home for Live Search of course.

The new name reminds me of the show Friends. Also, I hope they get a new favicon - I don't enjoy the stretched 'b' nor its color scheme.

microsoft technical domain history search archive dns bing

Evil Mad Scientist Laboratories - How to make a Sawed-off USB Key

2008 Jul 17, 5:39DIY instructions on making a flash drive that looks like a cut up USB cable: "Holy crap-- somebody just went and TORE MY FREAKING USB CABLE IN HALF while it was still attached to my laptop!!! No-- wait-- sorry. That's just my USB drive. My bad. Never mind

diy humor flash usb tutorial

Bluelounge - The Sanctuary

2008 Jun 9, 4:44"The Sanctuary is a beautiful, simple solution to a real everyday problem. A place to put the multitude of personal items we all carry around so they are easily located again when later needed and, always fully charged"

gift gadget design cable battery phone mp3player charger shopping product

GPS Stolen

2008 Jun 6, 3:03

My GPS was stolen last night or this morning and I'm missing it already. For instance when I drove to Novus glass repair to get my front passenger window replaced I drove down the wrong road for a while.

When I got out of my apartment this morning there was a police car sitting in my parking lot and the officer asked me: "David? ... What'd you leave in your car?". My face must have changed a lot when I had the following sequence of realizations: (a) a police officer is asking for me by name, (b) I'm not in trouble, (c) my car must have been burgled, and (d) my GPS must be stolen.

The officer was waiting outside my complex because someone had reported my car's broken window to the police in the morning. The officer was very courteous and upon taking my date of birth noted that we were born on exactly the same day. The window's safety glass was shattered and lying in tons of tiny pieces all over the passenger seat, my glove box was open and the middle armrest where I keep my CDs was open. Nothing appears to be missing other than the GPS, the GPS power cable, and the GPS dash mount. Adding insult to theft, the their scattered my CDs throughout my car and didn't take any of them, insulting my taste in music.

My car's window should be repaired now and hopefully the rain that came in through the broken window until I covered it with plastic bags (classy!) didn't do any permanent damage.

gps theft personal nontechnical