JS+Django: streaming multi-part ajax responses (MXHR)

Ajax is great eh? What did it stand for again…? Asynchronuous Javascript And XML?

Anyone actually sending back XML in their Ajax app these days?

I find, if I have the choice, I’m using JSON for sending the data back. Then, for those times where you don’t want to duplicate all of the templating logic from the server in your Javascript code just to ‘Ajax’ a form or something, I find I will often render a bit of HTML and send that back, even if it feels a bit dirty. It’s easy and there’s a certain DRYness that’s undeniable.

So it’s only a matter of time before you start thinking, as a did, “Sure, I could make half a dozen separate XHR requests, but what I really need to do here is send back a package of identifiable HTML and JSON chunks.”

Some kind of multi-part response basically. Hang on, I’ve heard of that before… something to do with email, or file uploads?

A quick search reveals some talk about how to do a multi-part form post via Ajax: basically it’s not possible, the XHR object doesn’t support it. That’s not what we want though… we want to parse multi-part data in the response.

It turns out all the browsers, except (of course) IE6, will give you access to the response body while the response is still loading. The multi-part format is pretty simple, each part is just separated by a delimiter string (see the IETF RFC here). So, we can write some js to parse the multi-part response and make it happen!

Well, actually I figured someone must have done it already.

I found this one first: mpAjax

It does the job, but doesn’t exactly follow the RFC. The script requires the boundary delimiter to have a random id appended to the end, where as the RFC seems to specify that the delimiters should all be identical. The RFC also provides a mechanism for specifying a different mime type for each part, with mpAjax your script will need to know in advance which mime type to expect for each part.

Bingo! DUI Stream

The other thing with mpAjax is that it doesn’t try and process each part as they arrive, it waits for the whole response before parsing. Well for the team at Digg it seems the ‘streaming’ aspect was the main point. They have an interesting demo where even stream down dozens of small (Base64-encoded…) GIF images via ajax, all in one request.

All good, they do streaming and they stick close to the RFC, allowing you to specify the mime type for each part. They put it up on GitHub in April ’09 promising a few enhancements and some more documentation would be forthcoming, but nothing has happened publicly since then.

After using the code a bit I took it on myself to fork and implement a few changes, such as: make the request via either GET or POST, make the request more than once (!), let it work – albeit without streaming – in IE6, and the ability to parse your own custom headers for each part.

Code is here on GitHub.

Django…

So, how do you output a multi-part http response from Django?

Glad you asked. Here’s four solutions (hacked out of a recent project I worked on)… all of which sub-class the django HttpResponse.

First up is a kind of base MultipartHttpResponse. Next is a sub-class of that which works with mpAjax.

Third is a sub-class that does true streaming, via a generator in the _make_body method. This is built-in django goodness, though be careful as I’ve read that some middleware modules can negate the streaming behaviour (they will consume the whole generator before passing on any output). This one will work with DUI Stream.

To use it, pass a list of (mime-type str, body) tuples as parts when initialising the response, eg:

parts = [
    (MIME_HTML,
     """<div>response</div>"""),
    (MIME_JSON,
    json.dumps(data)),
]

return StreamingMultipartHttpResponse(parts)

Easy.

Finally, there’s a sub-class of the above designed to take advantage of my tweaked version of the DUI Stream js. Code is as above, except that it takes a dict of headers in place of the mime-type string above, eg:

parts = [
    ({
        'Content-Type': MIME_HTML,
        'Content-ID': '%s|%s' % (i, x),
    },
     """<div>response</div>"""),
    ({
        'Content-Type': MIME_JSON,
        'Content-ID': '%s|%s' % (i, x+1),
    },
    json.dumps(data)),
]

return DUI_MXHR_Response(parts)

Cool?

You can check the DUI examples for the js side, but the code is nice and simple. You can register a handler function for each mime-type, which will fire whenever a part with that type has finished streaming down, eg:

    dui_stream_obj = new DUI.Stream();

    dui_stream_obj.listen('application/json', handleJson);
    dui_stream_obj.listen('text/html', handleHtml);
    dui_stream_obj.listen('complete', streamCheckComplete);

Nice.

If you’re using my version then your handler function will receive an array of headers for that part, eg:

function handleHtml(data, headers) {
    var path = headers['Content-ID'];
    ...

Done.

Advertisements

About anentropic
songwriter, musician, and er web programmer...

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: