How do We Make Encryption More Accessible?

This post is about a program I wrote to publish public keys in gravatar images. You can try it (if you’ve got a gravatar account) here or look at the source here.

HELO
250 Hello, I am glad to meet you
MAIL FROM:˂professor.bob@staff.uni.edu˃

When I was at university, discovering that I could easily have a conversation with the computer that managed mail was great fun. There were rules for our conversation of course, but they all seemed very civilized; I had to start by saying HELO, and it would give me some equally polite response. After that though, it quickly became obvious that it was a little too trusting. If I said that I was giving it mail from the.president@theWhitehouse.gov, or some lecturer or even some cute girl that my friend liked and that the mail was for my friend, it would just say Ok, Bye and then go ahead and deliver it.

It doesn’t take a genius to see that there’s quite a lot of potential for mischief there. Things have moved on a little since then, but probably less than you think. Email for most people is still not much more advanced than passing crumpled notes around a classroom. It has many of the same features – anyone on the path from the originator to the recipient can read it, the originator can pretend they never sent it if it turns out that their affection is not reciprocated, or worse, it’s easy for someone to forge one and create all kinds of opportunity for comedy. I suppose the main difference is that instead of it being a classroom, we’re talking about the world, and instead of it being about whether Emma likes Richard or not (she did), it’s about every aspect of our lives, including money, job, family, etc.

PRISM logo

This has been in peoples minds recently with news that various governments are logging every word, not just of email, but of everything we do online. This of course opens up even more potential for mischief. Normal individuals have already used the no fly list as a way of punishing people they didn’t like, but given the NSA and others, a few well placed forged emails might be easier than the phone calls and less risky.

Anyway, I’ve got some great news. A bunch of guys have been working on this, and they’ve come up with a really clever system that makes it basically impossible for anyone you didn’t intend to look your messages to read them. I was telling my Dad about it recently, and he found it hard to believe, but it’s just a bunch of maths, not even as complicated as you might expect and it more or less completely solves this problem.

There is of course a catch. The catch is that this scheme was invented independently at least twice in the 1970s, predates the use of the word ‘internet’ yet for some reason normal people are still using systems that are much less deserving of trust than they appear.

One of the problems is that of ‘key management’. You usually use a pair of computer ‘keys’ to enable this kind of security and you need a way to make one as available as possible, and the other as secret as possible. This is not really within the realm of possibility for normal humans, who regularly lose their physical keys or ‘hide’ them in obvious places despite the fact that they secure a majority of their worldly possessions. That’s a simple metal token that can easily be carried everywhere with you and is always around when you need it, and can’t be copied quickly without you noticing (generally speaking). The computer keys are harder to change if stolen, and easier to lose. Basically I shouldn’t be trusted with anything as precious as my own private key.

The next problem is that so few people are currently using any better system that it becomes difficult to find people to communicate with. This is probably partially because of the third problem:

The third problem is probably just software. There is no software that doesn’t make sending and receiving encrypted messages a massive pain. On top of that it’s hard (but not impossible) to enable important features like search on encrypted data.

I came up with a few ideas of my own to try to make this more reasonable. I wanted to use images with public keys embedded into them, so that when you see the image of someone in your address book, you also have the means to send them private messages, and possibly little seal images that are your private key, so you drag them onto data to sign the data. I think a much more visual, drag and drop system stands a much better chance of getting people using it. I also wanted to build on top of services that people already know about. Bitcoin is an encryption based currency system that lots of people are interested in, and Gravatar provides a sort of directory of email addresses to images.

The perfect system that I imagine is something that is easy enough to use for people who aren’t very technical, builds on top of things like Bitcoin and Gravatar, and probably provides a mail service that can’t read your stored history of emails but allows you to search them quickly and automatically encrypts messages to people or alternatively sends them a link that enables them to retreive the message over https. I didn’t create that, but I have created a proof of concept to show that some of these ideas are pretty doable. It’s still very rough and ready, so it’s not something for the non technical yet, but I think it shows one approach we could try using to bring private communications to everyone. You can try it (if you’ve got a gravatar account) here or look at the source here.

Modern Times: A New Clock

If you’re a regular reader, you might know that I occasionally play with different ways of representing time. I’ve come up with my latest version which I call UIT, and you can see it on github.

Clock Difficulties

Why do we need a new clock? Well, you could ask William Burke. On the 31st of October 1828, William Burke and his co-conspirators murdered Mary Docherty, the last in a line of 16 murders they’d committed in order to sell the bodies to the medical profession. When the law finally caught up with them, Burke and his partner, Helen McDougal had already disposed of the body and agreed their story; Burke did indeed meet Mary Docherty in a pub and invited her back to stay with them, but she’d left before 7 and they had no idea where she’d gone. When the police questioned them separately, Burke insisted that she’d left at 7 in the morning while Helen swore she’d left at 7 in the evening.

It can be difficult to get your story straight when you’re dealing with time, but Burke and McDougal got off lightly. In September 1999, three terrorists died at the same time in two different cities. Both groups were killed by the bombs they were transporting which were set to go off at 17:30. The confusion was simply that the bombs were set to go off according to Palestinian Daylight Savings Time while the guys driving the cars had made their plans on Israel Standard Time.

Even if you’ve never found yourself hanged or blown up over a clock confusion if you’re anything like me you’ve still made plenty of mistakes with clock arithmetic and daylight savings time.

Origins

So why is the clock the way it is? Dividing a ‘day’ into 12 parts was something that the ancient Egyptians started. They liked to set their sundials up with a dawn and a dusk hour, and ten full daylight hours. Since it was based on the hours of sunlight, the length of an hour would change with the seasons. In 1500 BC, the Egyptians had already invented a portable shadow clock, the spiritual precursor to the modern watch.

The Egyptians were not alone in preferring to divide things into twelves. The Babylonians had a similar obsession inherited from the Sumerians. The Sumerians were one of a group of peoples who independently invented writing around 3500 BC. The Sumerians pretty much invented civilisation as we know it, but they had inherited the Wheel and a number of other important advances from a culture that was lost to climate change (the 5.9 kiloyear event), the Ubaid culture. Anyway, for the Sumerians, counting to 60 seems to have been as natural as counting to 10 is for us. One suggestion for how they may have done it is by using five fingers from one hand to point to one of the 12 finger joints on the other hand, which perhaps explains why they divided the whole day into 12 parts (each around 2 of our modern hours).

When you’re wondering when to get up, when to eat lunch and when to feed the animals, measuring time according to the amount of light left in the day is a good idea, but when you’re trying to formalise systems and work with people in different parts of the world it becomes awkward. The ancient Greeks decided that the Egyptian system of variable length daytime hours was no good, and started splitting the day into 24 equal length hours. This was quite a big change and was by no means universally accepted until the advent of mechanical clocks made it by far the easiest way to measure time.

But why have we kept such an ancient system? After all, the Babylonians tended to measure things based on the lengths of various body parts or the weight of a grain of Barley. We don’t do that any more, but we seem to have carried their predilection for 60s into an age where it’s pretty inconvenient.

A New System

When you’re trying to come up with a new system, nature throws a couple of roadblocks in the way. There are two extremely useful natural time periods – the solar day, which is a nicely predictable cycle of light and dark, and the solar year which is a larger cycle of weather and temperature. Unfortunately in absolute terms, both vary annoyingly making them unsuitable for use in science and engineering, and even worse, the solar year is not an integer multiple of the day.

I think that while science and engineering may need a fixed length period of time, perhaps based on a fraction of the speed of light, this is not particularly useful for day-to-day use. So I wanted to choose a humane system based on either the length of a year or the length of a day. Ultimately I choose day, because I find it more useful to have lunch at the same time every day than to buy a winter coat at the same time every year.

Astronomical Clock Face

The obvious choice therefore is to measure everything in fractions of a solar day. I’d already hit on this by accident – if you create time units based on tens and hundreds, what you are actually doing is just naming some of the different places in the fraction of a day decimal. The places that relate most closely to what we currently do are to have hour-equivalents which are tenths of a day, minute-equivalents which are hundredths of an hour-equivalent, and second-equivalents which are hundredths of a minute-equivalent. Interestingly, the second equivalents come out fairly close to old-style seconds, and calculations become incredibly easy – 8:30 plus 5:80 becomes 1 day 4:10 (or 0.83 + 0.58 = 1.41).

But solving my difficulties with Dr. Kawashima’s Brain Training was not the extent of my ambition. I’ve also had to work extensively with people in distant timezones, and it can be very confusing to know when you’re actually going to have a meeting, so I wanted a time that was numerically the same everywhere. We could do this already – everyone could use GMT for example, but that’s awkward, because it’s not obvious what time midnight or noon is where you are. Actually, it’s currently not obvious when noon is, because you probably don’t live at the exact center of your time zone. For example Oxford Time should be 5 minutes and 2 seconds behind Greenwich Time, but with the old system, this is too difficult to represent. These days, we can calculate this kind of thing exactly, so my new clock requires access to your GPS coordinates. While the numbers stay the same for everyone, their position on the clock face always ensures that your local solar noon is at the top of the clock face and solar midnight is at the bottom of the clockface, with high accuracy. Another benefit of having the whole day be one complete rotation of the clock is that I can draw on the dark and light times, based on your location with sunrise and sunset marked on every day.

Of course, choosing the day to have your IM meeting can be difficult too, since days start at different times for different people, so I’ve invented a new set of days, with a base 10 as well, going Nullday, Unday, Duoday, Triday, Quadday, Pentday, Hexday, Heptday, Octday, Nonday. These days form a new 10 day week. Now, I don’t want to replace the 7 day work week (well actually I do, I’d like 4 days on, 3 days off), so I’ve made sure my clock shows the old weekdays too, but when organising meetings, you should use the new weekdays. Instead of months, they get numbered in the year, so the first day of the year is the zeroth Nullday, and the last day of most years is the 36th Pentday, although some years will have a 36th Hexday.

Sadly, I can’t really fix the years, but once everyone has adopted this system of time, perhaps we can work on modifying the Earths orbit to make the years more sensible too.

In order to make the transition easy, I’ve made a canvas based app that should run in your browser (even on mobile devices), which is available here. It can hook in to Google Calendar and draw your meetings on the clock face, it displays sunrise and sunset and shows noon at the top and midnight at the bottom based on your location. It even shows you legacy time too, so that you can still talk to any luddites you might know.

But I can’t be responsible for bad outcomes if you make arrangements to plant bombs or evade justice with anyone still using the old time system.

SolarPosition

I’ve ported some C code designed to calculate the suns position into javascript. It’s all part of my plan to make a clock that shows the local sunrise and sunset times on its face.

Given a bewildering number of parameters (most of which you can just guess), it will tell you the time of sunrise, sunset and solar noon.

It also has a few methods for calculating time based on fractions of days, and the Julian Day (a commonly used system in astronomy).

You can try it out here.

SoleMirror

Particularly when I start a new web project, I often find myself wanting a javascript console that I can embed in a page. The consoles built into modern browsers are usually very good, but it’s also nice to have something useful for logging and visible on the screen as soon as the page is loaded.

As it happens, the excellent CodeMirror implements a lot of the functionality I needed and is easy to extend, so I added a few small pieces of javascript and ended up with SoleMirror, a CodeMirror javascript console.

You can use it at the SoleMirror console page or get the source at the github page. There’s even a bookmarklet that will install it into other pages.

This was just something that I put together over a couple of hours, so there are certainly inelegancies and I’ve not tested it in browsers that aren’t ‘fun’. Maybe you are just the person to improve it…

The Madness of the HTML5 FileSystem API

I’m quite excited about the possibilities that the FileSystem API opens up to web applications. At the moment it’s only implemented in Chrome, but the w3c has a spec and I hope to see it in other browsers soon. I’ve been working on a library to make it easier to use.

The FileSystem API is asynchronous, so you’ll notice quickly from the html5rocks guide that even simple tasks involve many many levels of nesting of callbacks to get them sequenced correctly. Take a look at this piece of code to create a file and then list all the files in the root folder:

window.webkitStorageInfo.requestQuota(PERSISTENT, 1024*1024, function(grantedBytes) {
  window.webkitRequestFileSystem(PERSISTENT, grantedBytes, onInitFs, errorHandler);
}, errorHandler);

function onInitFs(fs) {
  fs.root.getFile('log.txt', {create: true}, function(fileEntry) {
        fileEntry.createWriter(function(fileWriter) {

          fileWriter.onwriteend = function(e) {

                function listResults(entries) {
                    // finished!  Output the directory entries
                    console.log(entries);
                };

                var dirReader = fs.root.createReader();
                var entries = [];

                // Call the reader.readEntries() until no more results are returned.
              var readEntries = function() {
                 dirReader.readEntries (function(results) {
                  if (results.length < 1) {
                    listResults(entries.sort());
                  } else {
                    entries.push.apply(entries, results);
                    readEntries();
                  }
                }, errorHandler);
              };

              readEntries(); // Start reading dirs.
          };

          fileWriter.onerror = errorHandler;

          // Create a new Blob and write it to log.txt.
          var bb = new WebKitBlobBuilder();
          bb.append('Hello FileSystem API');
          fileWriter.write(bb.getBlob('text/plain'));

        }, errorHandler);
  }, errorHandler);
}

function errorHandler(err) {
    console.log("ERROR", err);
}

While this code can be tidied up a little, compare it with the bash commands to do the same thing:

echo Hello FileSystem API > log.txt
ls

There's quite a difference in length and complexity. You may not have noticed because some of it is split out, but by the time we get to the listResults function and the end of the task, we are 5 levels deep, mainly in closures.

Here's the equivalent code using this library:

var fs = FS.request(window.PERMANENT, 1024*1024);
fs.seq(
    FS.getFile("log.txt", true),
    FS.write("Hello FileSystem API"),
    fs,
    FS.list
).then(function(entries) {
    // we're done!
    console.log(entries);
}, function(err) {
    console.log("ERROR", err);
});

It's still not as simple as the bash code, but the nesting has been turned into sequential calls, and this gives us the power to compose and reuse pieces of functionality much more easily. It's actually an implementation of the Monad Design Pattern; Promises that gives us the ability to unwrap all of those nested calls.

Design Pattern: Wrapper with Composable Actions

Name And Classification

Wrapper with Composable Actions. An extension to the Wrapper pattern that allows operations to be composed on wrapped objects.

This pattern is well known in another context, but its usual name and presentation is not very descriptive so I’m avoiding it deliberately until the end.

Problem

An object has been placed in a wrapper but it must still be possible to use it as input to a chain of operations.

Context

As well as the common uses for wrappers (changing one interface to another), there are other situations where it’s useful to wrap objects, for example:

  • if you need to keep track of a result plus extra information about the actions you’ve carried out as you perform them.
  • if the value you are wrapping is not simple in some way – perhaps it is going to be computed asynchronously and the wrapper represents a promise to provide the result.
  • if you want to add decision points or extra actions based on data not included in the wrapped object, but available in the wrapper that should be done on every operation. (This is like the decorator pattern).
  • if you want to indicate that something fundamental has changed, you can use a wrapper object that doesn’t provide the ability to get at the original object.

Actions (operations that take a value) are usually created to operate on the nonwrapped versions of objects, which means that in the general case, wrapped objects can sometimes be difficult to compose actions on. This pattern specifies two functions that can be implemented to assist with composing actions on wrapped objects.

In general, this pattern is useful when considering a sequence of operations where each step depends to some extent on the previous value and produces a new value. Because of this it is commonly employed in functional languages. Its implementation in languages without first-class functions is generally awkward. This pattern can provide extra value in languages with a strong (stronger than java) type system.

Solution

To ensure composability, two functions need to be provided, wrap and chain (although their exact names can vary based on situation). I have shown them here on the interface Wrapper:

/* This is analogous to an operation that takes something of type T and returns something of type A, except
 * that A is provided in wrapped form.
 * In a language with first class functions, Action describes a function rather than an object interface
 *    i.e. T -> Wrapper< A >
 */
interface Action< T, A > {
	Wrapper< A > call(T value)
}

interface Wrapper< T > {
	// This is likely to be provided as a static method or constructor.
	Wrapper< T > wrap(T value);

	/* This is normally more complex than simply returning the result of the action, as there
	 * is likely to be some behaviour or state embedded in *this* Wrapper< T > that needs to be appropriately combined
	 * with the Wrapper< A > that is got by executing the action.
	 */
	< A > Wrapper< A > chain(Action< T, A > action);
}

Example 1: Promises

Promises are wrappers for value objects that may not yet be available for calculation. Promises are typically instances of this pattern. Here is a (much simplified) example in javascript:

function Promise() {
	this.onResolve = [];
};
Promise.prototype.chain = function(thenFunc) {
	// return a new promise that will resolve at the same time
	// and with the same value as the result of the passed function
	var result = new Promise();
	this.onResolve.push(function(val) {
		var resultingPromise = thenFunc(val);
		resultingPromise.chain(function(val) {
			result.resolve(val);
		});
	});
	return result;
};
Promise.prototype.resolve = function(value) {
	// tell all functions waiting for a value that we have one
	this.onResolve.forEach(function(then) {
		then(value);
	});
	// clear down (to avoid memory leaks)
	this.onResolve = null;
	// from now on, chain gets called immediately with the value.
	this.chain = function(then) {
		return then(value);
	}
};
Promise.wrap = function(value) {
	var result = new Promise();
	result.resolve(value);
	return result;
};

This can then be used like so:

function parseJSON(json) {
	return Promise.wrap(JSON.parse(json));
};

function getURL(url) {
	var result = new Promise();
	var xhr = new XMLHttpRequest();
	xhr.onreadystatechange = function() {
		if (xhr.readyState == 4 && xhr.status == 200) {
			result.resolve(xhr.responseText);
		}
	}
	xhr.open("GET", url, true);
	xhr.send();
	return result;
}

function postStatistics(obj) {
	var xhr = new XMLHttpRequest();
	xhr.open("POST", "http://statserver", true);
	xhr.send(obj.stats);
	return Promise.wrap();
};

Promise.wrap("http://myserver/data.json")
	.chain(getURL)
	.chain(parseJSON)
	.chain(postStatistics);

The big win here is that actions that would otherwise have had to be nested can be composed into a data processing chain, and synchronous and asynchronous actions can be composed together transparently.

Example 2: History

Supposing you want to store the steps taken in a computation as you go along. You can use this pattern to achieve this:

function History(val) {
	this.val = val;
	this.commands = [];
}
History.prototype.chain = function(action) {
	var result = action(this.val);
	result.commands = this.commands.concat(result.commands);
	return result;
};
History.wrap = function(val) {
	return new History(val);
};
History.prototype.toString = function() {
	return this.commands.join("\\n")+"\\n"+this.val;
};

With some actions defined:

function addTen(val) {
	var result = History.wrap(val + 10);
	result.commands.push("added ten");
	return result;
}

function divideTwo(val) {
	var result = History.wrap(val / 2);
	result.commands.push("divided by two");
	return result;
}

You can now do a calculation and then print out the result and the steps taken to achieve it like so:

History.wrap(8).chain(addTen).chain(addTen).chain(divideTwo).chain(addTen).toString()

This time, extra information was threaded through the whole process, but the individual steps in the chain remained composable.

Extensions

Often there are already a body of functions that operate on unwrapped values, returning unwrapped values. An ‘apply’ function can be defined in terms of the two functions already added to allow them to be easily used:

Wrapper.prototype.apply = function(func) {
	return this.chain(function(val) {
		return Wrapper.wrap(func(val));
	});
};

Big Reveal

As you may have realised already, this pattern is also known as ‘Monad’ in functional programming, where the wrap function is called ‘unit’ (or ‘return’) and the ‘chain’ function is called ‘bind’ (or >>=).

Despite the fact that Monads are commonly associated with Haskell and functional programming, they’re used and useful in many places and I think describing them as a Design Pattern is much more accessible than describing them by their similarity to category theory constructs.

I came to this conclusion after realising that I needed promises to make a reasonable version of the javascript FileSystem API (which is available on github) and then suddenly realising that Promises are actually Monads, something that nobody had mentioned to me before.

Links:

Javascript Puzzlers

Here are a few javascript puzzlers. If you want to try them yourself, all of the statements below evaluate to true and you have to work out what values could make that happen…

Starting simply, name a value that is not === to itself:

    a !== a

And its dual – name two values that are different but do === each other. The second part of this puzzle is to provide a statement that can distinguish b and c. (clue – don’t look unless you’re stuck)

    b === c

What’s a value that seems to be the same as its complement (there are quite a few possibilities):

    !d == d

How about something that seems to give 42 more than you might expect (Can you find a second way of doing the same thing)?

    (e + 4 === 42) && ("4" + e.toString() === "42")

How did you do?

Playing on the CouchDB

Let’s say you’re playing with CouchDB and javascript. It’s great fun, but you’re in development mode, you want to be able to change a file and then reload in your browser without having to mess around uploading and deploying stuff.

The ‘proper’ way of doing this is probably to set up a reverse proxy so that your couchdb is visible through your development webserver, and all is good, but that sounds like a lot of configuration for a playing around kind of setup.

What you’d like to do is run a normal couchdb install on your machine, and use whatever you normally do for webserving (for just playing around, I tend to use a jetty jar I’ve created as it’s very light and easy, but you could equally use tomcat or whatever), and edit the javascript and html files in place. If you have this set up, you can’t usually load the javascript couchdb api from the couchdb server, since it uses an XHR to communicate with the couchdb server, and these are restricted to using exactly the host that the page is loaded from. The fact that the couchdb server is on a different port to your webserver stops it from working.

You can get around this by saving a simple html document as an attachment in your couchdb (any database). This html document is then loaded in an iframe, sets the document.domain to the hostname (removing the port), and then it passes the api object up to the parent window.

If you’ve attached that file to a document with id ‘document’ in a database called ‘webaccess’, you would then use it as below:

Advent Candles

A while ago I made some candles for my wife (I know! How romantic!). Since it’s advent, I thought I’d share them with you all. They have a little easter egg that only works on certain machines and browsers. Anyone care to guess / work out what it is without cheating and looking in the code?

Direct link.

Tetravex

I’ve recently been playing a puzzle game called tetravex on Ubuntu, so I thought I’d write a version of it for HTML5. It’s designed to work on android phones and the iphone (only in vertical orientation at the moment), but if you’re using a modern browser you should see a working version of it below. The GUI still needs a lot of work, and I’m sure there are bugs, but pretty much all the functionality of the linux version is in there. On a phone, you can visit this direct link and then save it onto your homescreen to run it as an application. This should let it run offline too. It uses localStorage to store the high score table.

Update: Turns out it uses some features that Apple don’t support on iphone yet. I’ll try to make it more backwards compatible later. In the meantime, Chrome, Desktop Safari, Firefox and Android Browser should all work fine.