SocketTornad.IO

Brendan W. McAdams [email protected]

Contributors

Matt Swanson

Implementation of the Socket.IO Websocket emulation protocol in Python on top of the non-blocking Tornado Web Framework. Socket.IO is a JavaScript library for providing full emulation of Websockets for browsers which don't support it. While the client-side programmer codes as if they have a constantly open bi-directional communication channel, Socket.IO will (if the browser doesn't support Websockets) use several fallback protocols to provide the behavior. These fallback protocols require a negotiation between the client and server to determine an agreeable protocol; the reference implementation of the server is done in Node.JS which was less than agreeable to our needs. There are also implementations in Ruby Rack and Go but we rejected those for simlar reasons to Node.JS.

This version is designed for making Pythonistas happy.

Implementing SocketTornad.IO

As a user your only major requirement is to subclass tornad_io.socket_io.SocketIOHandler. This base class provides Tornado Handler logic for the entire Socket.IO protocol - it automatically responds to protocol handshakes and notifies you of three events, represented by Python methods on your class:

1. on_open: Called when a Socket.IO handshake completes successfully and a client is brought online. Gets a copy of the *args and **kwargs from the request... can be used for you as a user to do further authentication of the connection. By way of example, we lookup certain authorization information once a connection finishes and decide if we'll allow the connection to continue. This is not a required method - you need not implement it if you don't care about it.
2. on_close: Called when a Socket.IO connection is fully closed. Passes no arguments, but lets you do any cleanup of database handles, etc. This is not a required method - you need not implement it if you don't care about it.
3. on_message: The main method. This is invoked whenever the browser client sends a message. It is automatically decoded, and any JSON will be passed as a fully parsed Python object. This method receives a single argument of message which contains the parsed message. You can respond with the self.send method (see below)

You can send messages to the client by use of the self.send method. This takes a single argument of message and transmits it to the client. If you pass a string it will be pased "as is" to the browser; if you want to send JSON you should pass a dict in, which will be JSON encoded and marked as JSON in the Socket.IO wire format. An Object is also acceptable as long as simplejson is able to encode it to JSON.

There is fallback code for the JSON import - if you don't have simplejson installed it will import the json module (based on simplejson) which has been included with Python since 2.6 instead (thanks to swanson for the patch). However, the version of json which ships with Python lacks built in support for encoding decimal.Decimal objects, which is why we prefer (as specified in setup.py) simplejson >= 2.1. If you do not have an appropriate version of simplejson installed and try to send an object or dict containing decimal.Decimal instances to the client, you may encounter errors.

For those of you who know Tornado already, do not call the self.write method unless you want things to act weird. self.write still (in the current iteration) sends raw data to the client - but Socket.IO uses a wire format which requires certain encoding. Anything you pass via self.write will likely not be understood by the client.

This is an example handler:

class EchoHandler(SocketIOHandler):
    def on_open(self, *args, **kwargs):
        logging.info("Socket.IO Client connected with protocol '%s' {session id: '%s'}" % (self.protocol, self.session.id))
        logging.info("Extra Data for Open: '%s'" % (kwargs.get('extra', None)))

    def on_message(self, message):
        logging.info("[echo] %s" % message)
        self.send("[echo] %s" % message)

    def on_close(self):
        logging.info("Closing Socket.IO Client for protocol '%s'" % (self.protocol))

This handler is meant to be simple: It merely echoes back any message it receives to the client. Were you to test this in your browser your console will reflect back what you send:

> socket.send("OMG! Ponies!")
[echo] OMG! Ponies!

(In this case I have my test page set to print any messages to console.log().)

Useful properties

Every subclass of SocketIOHandler has a few useful properties attached to it:

• protocol: This is a string containing the name of the protocol currently being used to communicate with the client.
• session: This is a Beaker Session object which can be used to track information about the connection in question. We use it internally to direct output to the right place in polling. Feel free to save your own data - just make sure to call self.session.save() if you modify it or your changes will be lost.

Configuring SocketTornad.IO

Routes (e.g. how clients access you)

It is necessary in Tornado set up your 'resources' (known in some frameworks as 'routes') which define what paths are valid and what controllers handle the request. Because the paths Socket.IO uses to negotiate the connection and speak (esp. in the fallback protocols) are hairy, we have instead created a ClassMethod on SocketIOHandler which allows you to easily get the correct route for your service:

echoRoute = EchoHandler.routes("echoTest", "(?P<sec_a>123)(?P<sec_b>.*)", extraSep='/')

This returns the data structure which Tornado expects to configure itself, with all requests for '/echoTest' pointed at the EchoHandler class. For the curious, the structure returned looks like this:

('/(?P<resource>echoTest)/(?P<extra>(?P<sec_a>123)(?P<sec_b>.*))/(?P<protocol>(websocket|xhr-multipart|htmlfile|jsonp-polling|flashsocket|xhr-polling))/?(?P<session_id>[0-9a-zA-Z]*?)/?((?P<protocol_init>\\d*?)|(?P<xhr_path>\\w*?))/?(?P<jsonp_index>\\d*?)', <class 'tornad_io.EchoHandler'>)

Hence the routes classmethod to easily configure with... resource can be any valid string, including, if necessary, a Regular Expression. Any requests beginning with your resource parameter will be routed to EchoHandler for processing. The additional cruft in there are regular expressions to handle the myriad of extra path information Socket.IO ships to find a valid protocol.

We also accept two additional optional parameters to configure routes (only the resource parameter is required).

• extraRE is an optional string containing a regular expression for 'extra' information to capture on the URL. In my case, I have a PHP process pass an authenticated secure token to the Tornado process on each request to help identify and authorize a user. I pass this as part of the Socket.IO resource - while my Tornado resource is configured as 'echoTest', I want to capture and separate the additional secure token. By setting up an extraRE SocketTornad.IO will automatically save the extra data in **kwargs['extra'] - specifically accessible in on_open for further authentication. You MAY put capture groups inside extraRE - if you name them they are available from on_open in **kwargs, otherwise they will be in unnamed buckets inside of *args*.

• extraSep indicates a character to separate the 'base' resource and extraRE with. By default there is none - they are expected to run together. I typically set extraSep to a '/' character.

After that you simply need to pass the configured route to Tornado:

application = tornado.web.Application([
    echoRoute
])

Service Settings (ports, etc)

You can configure the service very easily by passing arguments into the Tornado application object. There are currently 4 user configurable properties:

• enabled_protocols: This is a list of the Socket.IO protocols the server will respond to requests for. Clients try them one by one until the server and client both find one they both support. The possibilities are:

• websocket: Standard HTML5 Spec Websockets. Our implementation uses the one built into Tornado with a slight tweak to message receipt to enable decoding of the special Socket.IO wire encoding format. (Works in Chrome and any other browser with native Websocket support)
• flashsocket: HTML5 Websockets emulated in Flash for older browsers like Firefox. EXACTLY the same implementation wise to websocket, but starts up a Flash policy server which is necessary for Flash sockets to work. (Tested in IE8, and Firefox 3)
• xhr-multipart: XMLHTTPRequest (AJAX) Multipart messaging. Opens and long polls a GET request to send from server to client, client sends a POST to send client to server. Uses multipart & chunking to send a continuous stream of messages down the same open GET channel. Best option after websocket/flashsocket. (Tested in IE8, Firefox 3 and Chrome)
• xhr-polling: XMLHTTPRequest (AJAX) Long Polling. Client polls on a GET until a message is available, closes the GET after getting a message and then opens a new one until a message is available. (Tested in IE8, Firefox 3 - does NOT work with Chrome at all)
• jsonp-polling: Identical protocol to xhr-polling but pushes Javascript script data via JSONp. (Tested in IE8, Firefox 3 - does NOT work with Chrome at all)
• htmlfile: Appears to be for much older IE browsers w/o proper AJAX support, creates an AJAX HTMLFile control and does some iframe nastiness. I haven't found a browser that properly supports this so if you test it let me know ... Copied implementation from reference Node code.

The default setting is to enable ALL protocols, i.e.:

    ['websocket', 'flashsocket', 'xhr-multipart', 'xhr-polling', 'jsonp-polling', 'htmlfile']

• socket_io_port: The port for the Socket IO Server to listen on. This configuration setting is ignored unless you explicitly use the SocketIOServer class to start Tornado (See below).
• flash_policy_file A fully qualified path to a Flash Policy XML File. A default permissive one is included in this distribution as flashpolicy.xml; by default the Flash service looks for flashpolicy.xml in the same directory as the current execution. This configuration setting is ignored unless you explicitly use the SocketIOServer class to start Tornado (See below).
• flash_policy_port The port for the Flash policy server to listen on. This defaults to port 843 - Flash absolutely will not connect to any other port so if you change this, make sure you setup a portmap on the frontend. Without a valid policy service Flash fallback sockets will not work. This configuration setting is ignored unless you explicitly use the SocketIOServer class to start Tornado (See below).

Configuring these settings is done by passing them to the tornado.web.Application constructor as kwargs:

application = tornado.web.Application([
    echoRoute 
], enabled_protocols=['websocket', 'flashsocket', 'xhr-multipart', 'xhr-polling'],
   flash_policy_port=8043, flash_policy_file='/etc/lighttpd/flashpolicy.xml', socket_io_port=8888)

Starting Up

Best Way: SocketIOServer

The SocketTornad.IO distribution contains a modified version of the Tornado HTTPServer class designed to automatically read the necessary configuration settings and start everything up. If flashsocket is enabled it will start the Flash Policy server, and it starts the Socket.IO Service for you (as opposed to you starting it up manually).

Assuming you set the configuration options on your Application instance (or are happy with the defaults) you need merely instantiate a tornad_io.SocketIOServer:

if __name__ == "__main__":
    socketio_server = SocketIOServer(application)

Starting Manually

If you'd like more control over how you start everything up you can start things manually, similar to the Tornado Docs. This requires booting the IOLoop yourself:

if __name__ == "__main__":
    flash_policy = tornad_io.websocket.flash.FlashPolicyServer(port=8043, policy_file="/etc/lighttpd/flashpolicy.xml")
    http_server = tornado.httpserver.HTTPServer(application)
    http_server.listen(8888)
    tornado.ioloop.IOLoop.instance().start()

Examples

Chatroom Example

There is a chatroom example application contributed by swanson. It is in the examples/chatroom directory. For instructions, please see its README.