The Madness of the HTML5 FileSystem API

I’m quite excited about the possibilities that the FileSystem API opens up to web applications. At the moment it’s only implemented in Chrome, but the w3c has a spec and I hope to see it in other browsers soon. I’ve been working on a library to make it easier to use.

The FileSystem API is asynchronous, so you’ll notice quickly from the html5rocks guide that even simple tasks involve many many levels of nesting of callbacks to get them sequenced correctly. Take a look at this piece of code to create a file and then list all the files in the root folder:

window.webkitStorageInfo.requestQuota(PERSISTENT, 1024*1024, function(grantedBytes) {
  window.webkitRequestFileSystem(PERSISTENT, grantedBytes, onInitFs, errorHandler);
}, errorHandler);

function onInitFs(fs) {
  fs.root.getFile('log.txt', {create: true}, function(fileEntry) {
        fileEntry.createWriter(function(fileWriter) {

          fileWriter.onwriteend = function(e) {

                function listResults(entries) {
                    // finished!  Output the directory entries
                    console.log(entries);
                };

                var dirReader = fs.root.createReader();
                var entries = [];

                // Call the reader.readEntries() until no more results are returned.
              var readEntries = function() {
                 dirReader.readEntries (function(results) {
                  if (results.length < 1) {
                    listResults(entries.sort());
                  } else {
                    entries.push.apply(entries, results);
                    readEntries();
                  }
                }, errorHandler);
              };

              readEntries(); // Start reading dirs.
          };

          fileWriter.onerror = errorHandler;

          // Create a new Blob and write it to log.txt.
          var bb = new WebKitBlobBuilder();
          bb.append('Hello FileSystem API');
          fileWriter.write(bb.getBlob('text/plain'));

        }, errorHandler);
  }, errorHandler);
}

function errorHandler(err) {
    console.log("ERROR", err);
}

While this code can be tidied up a little, compare it with the bash commands to do the same thing:

echo Hello FileSystem API > log.txt
ls

There’s quite a difference in length and complexity. You may not have noticed because some of it is split out, but by the time we get to the listResults function and the end of the task, we are 5 levels deep, mainly in closures.

Here’s the equivalent code using this library:

var fs = FS.request(window.PERMANENT, 1024*1024);
fs.seq(
    FS.getFile("log.txt", true),
    FS.write("Hello FileSystem API"),
    fs,
    FS.list
).then(function(entries) {
    // we're done!
    console.log(entries);
}, function(err) {
    console.log("ERROR", err);
});

It’s still not as simple as the bash code, but the nesting has been turned into sequential calls, and this gives us the power to compose and reuse pieces of functionality much more easily. It’s actually an implementation of the Monad Design Pattern; Promises that gives us the ability to unwrap all of those nested calls.

Design Pattern: Wrapper with Composable Actions

Name And Classification

Wrapper with Composable Actions. An extension to the Wrapper pattern that allows operations to be composed on wrapped objects.

This pattern is well known in another context, but its usual name and presentation is not very descriptive so I’m avoiding it deliberately until the end.

Problem

An object has been placed in a wrapper but it must still be possible to use it as input to a chain of operations.

Context

As well as the common uses for wrappers (changing one interface to another), there are other situations where it’s useful to wrap objects, for example:

  • if you need to keep track of a result plus extra information about the actions you’ve carried out as you perform them.
  • if the value you are wrapping is not simple in some way – perhaps it is going to be computed asynchronously and the wrapper represents a promise to provide the result.
  • if you want to add decision points or extra actions based on data not included in the wrapped object, but available in the wrapper that should be done on every operation. (This is like the decorator pattern).
  • if you want to indicate that something fundamental has changed, you can use a wrapper object that doesn’t provide the ability to get at the original object.

Actions (operations that take a value) are usually created to operate on the nonwrapped versions of objects, which means that in the general case, wrapped objects can sometimes be difficult to compose actions on. This pattern specifies two functions that can be implemented to assist with composing actions on wrapped objects.

In general, this pattern is useful when considering a sequence of operations where each step depends to some extent on the previous value and produces a new value. Because of this it is commonly employed in functional languages. Its implementation in languages without first-class functions is generally awkward. This pattern can provide extra value in languages with a strong (stronger than java) type system.

Solution

To ensure composability, two functions need to be provided, wrap and chain (although their exact names can vary based on situation). I have shown them here on the interface Wrapper:

/* This is analogous to an operation that takes something of type T and returns something of type A, except
 * that A is provided in wrapped form.
 * In a language with first class functions, Action describes a function rather than an object interface
 *    i.e. T -> Wrapper
 */
interface Action {
	Wrapper call(T value)
}

interface Wrapper {
	// This is likely to be provided as a static method or constructor.
	Wrapper wrap(T value);

	/* This is normally more complex than simply returning the result of the action, as there
	 * is likely to be some behaviour or state embedded in *this* Wrapper that needs to be appropriately combined
	 * with the Wrapper that is got by executing the action.
	 */
	 Wrapper chain(Action action);
}

Example 1: Promises

Promises are wrappers for value objects that may not yet be available for calculation. Promises are typically instances of this pattern. Here is a (much simplified) example in javascript:

function Promise() {
	this.onResolve = [];
};
Promise.prototype.chain = function(thenFunc) {
	// return a new promise that will resolve at the same time
	// and with the same value as the result of the passed function
	var result = new Promise();
	this.onResolve.push(function(val) {
		var resultingPromise = thenFunc(val);
		resultingPromise.chain(function(val) {
			result.resolve(val);
		});
	});
	return result;
};
Promise.prototype.resolve = function(value) {
	// tell all functions waiting for a value that we have one
	this.onResolve.forEach(function(then) {
		then(value);
	});
	// clear down (to avoid memory leaks)
	this.onResolve = null;
	// from now on, chain gets called immediately with the value.
	this.chain = function(then) {
		return then(value);
	}
};
Promise.wrap = function(value) {
	var result = new Promise();
	result.resolve(value);
	return result;
};

This can then be used like so:

function parseJSON(json) {
	return Promise.wrap(JSON.parse(json));
};

function getURL(url) {
	var result = new Promise();
	var xhr = new XMLHttpRequest();
	xhr.onreadystatechange = function() {
		if (xhr.readyState == 4 && xhr.status == 200) {
			result.resolve(xhr.responseText);
		}
	}
	xhr.open("GET", url, true);
	xhr.send();
	return result;
}

function postStatistics(obj) {
	var xhr = new XMLHttpRequest();
	xhr.open("POST", "http://statserver", true);
	xhr.send(obj.stats);
	return Promise.wrap();
};

Promise.wrap("http://myserver/data.json")
	.chain(getURL)
	.chain(parseJSON)
	.chain(postStatistics);

The big win here is that actions that would otherwise have had to be nested can be composed into a data processing chain, and synchronous and asynchronous actions can be composed together transparently.

Example 2: History

Supposing you want to store the steps taken in a computation as you go along. You can use this pattern to achieve this:

function History(val) {
	this.val = val;
	this.commands = [];
}
History.prototype.chain = function(action) {
	var result = action(this.val);
	result.commands = this.commands.concat(result.commands);
	return result;
};
History.wrap = function(val) {
	return new History(val);
};
History.prototype.toString = function() {
	return this.commands.join("\n")+"\n"+this.val;
};

With some actions defined:

function addTen(val) {
	var result = History.wrap(val + 10);
	result.commands.push("added ten");
	return result;
}

function divideTwo(val) {
	var result = History.wrap(val / 2);
	result.commands.push("divided by two");
	return result;
}

You can now do a calculation and then print out the result and the steps taken to achieve it like so:

History.wrap(8).chain(addTen).chain(addTen).chain(divideTwo).chain(addTen).toString()

This time, extra information was threaded through the whole process, but the individual steps in the chain remained composable.

Extensions

Often there are already a body of functions that operate on unwrapped values, returning unwrapped values. An ‘apply’ function can be defined in terms of the two functions already added to allow them to be easily used:

Wrapper.prototype.apply = function(func) {
	return this.chain(function(val) {
		return Wrapper.wrap(func(val));
	});
};

Big Reveal

As you may have realised already, this pattern is also known as ‘Monad’ in functional programming, where the wrap function is called ‘unit’ (or ‘return’) and the ‘chain’ function is called ‘bind’ (or >>=).

Despite the fact that Monads are commonly associated with Haskell and functional programming, they’re used and useful in many places and I think describing them as a Design Pattern is much more accessible than describing them by their similarity to category theory constructs.

I came to this conclusion after realising that I needed promises to make a reasonable version of the javascript FileSystem API (which is available on github) and then suddenly realising that Promises are actually Monads, something that nobody had mentioned to me before.

Links: