Archive
More comet with gevent
Check out http://rss.im/stream/
This is my response to Superfeedr competition. It’s a gevent/django webapp that reads an Atom stream provided by Superfeedr, converts the entries to JSON and broadcasts them to a number of long polling listeners, similar to the long polling chat example I posted before.
On the client side, to make the stream digestible by humans, scrolling is avoided altogether and the pace of the stream is controlled by hovering mouse over the entries.
The source code: http://bitbucket.org/denis/stream-web/
Simpler long polling with Django and gevent
Recently released Tornado web server includes an example chat application. This post describes a modification of that example that runs on Django and gevent wsgi server. The modified version achieves the same goal while staying within a familiar web framework Django and using simpler concurrency model.
- live example is running at http://chat.gevent.org/
- example code at bitbucket
It implements a simple web chat room with instant notifications and does so by using Ajax with long polling. A dynamic web application of this kind is thought of as a better fit for an asynchronous framework like Tornado or Twisted, than for a traditional thread-pool and/or process-pool based approach like apache+mod_wsgi. This is because each user participating in the chat (or simply not closing their browser window) maintains an open connection with the server for message updates and the amount of memory an open connection takes on server is significantly different depending on the server setup: a few KB for async versus a few MB for thread/process.
However, using an asynchronous framework requires twisting your code somewhat: because it’s all running in the same thread, you cannot simply wait in a view handler until a new message is available, then construct a response and return it to the framework. Instead, you usually return a callback that will be called when a new message is posted to generate the response and thus complete the long polling request. Not a huge obstacle but it does obscure the code flow.
If only we had more light-weight units of execution than threads and processes, implementing ajax apps like this chat would be a lot simpler. Turns out we do, and there are options: Stackless Python and greenlet. The latter is an extension module that runs on the stock Python and that’s what gevent currently supports.
The wsgi server bundled with gevent creates a new greenlet for each incoming connection making it’s possible in a request handler to sleep, wait for event and even access network without blocking anyone. Greenlets are cheap, memory-wise, so it’s about as scalable as callback- or Deferred- based solution. The logic, however, becomes much more transparent:
- When a new message is posted, set the event.
- When a client requests the updates (and it already has the latest message), wait for the event.
The code: views.py
Concurrency in Python 