Evergreen: cooperative multitasking and i/o for Python

Posted on by saghul 2 Comments

I’ve been working on-and-off on this project for almost a year during my free time, and after meditating about it I thought: “fuck it, ship it”. Allow me to introduce Evergreen: cooperative multitasking and i/o for Python.

“So, another framework?” I hear you say. Yes, it’s another async framework. But it’s my async framework. I’ve used a number of frameworks for developing servers in Python such as Twisted, Tornado, Eventlet, Gevent and lately Tulip and all of them have great and not so great things, so I decided to blend the ideas I gathered from all of them, add some opinionated decisions, some Stackless flavour and Evergreen was the result.

standards

Evergreeen is a framework which allows developers to write synchronous looking code which is executed asynchronously in a cooperative manner. Evergreen presents an API which looks like the one you would use to write concurrent programs using threads or futures from the Python standard library. The facilities provided by Evergreen are however cooperative, that is, while a task is busy waiting for some i/o other tasks will have their chance to run.

“Show me the code!” I hear you say. Sure, it’s up here on GitHub, released under the MIT license. Since the usual example is a web crawler, here you have one.

Did I mention it supports Python 2 and 3?

“Is it production ready?” I hear you say. It’s still on a very early stage, but I believe the foundation is solid. However, the APIs provided by Evergreen may change a bit until I feel confortable with them. All feedback is welcome, so if you give it a try do let me know!

I’d like to thank all authors of similar libraries for releasing their work as Open Source which I could look into and learn from.

I hope Evergreen can help you solve some problems and you enjoy using it as much as I do developing it.

:wq

 

Interactive Python interpreter over TCP for Tornado applications

Posted on by saghul Leave a comment

The Python standard library never ceases to surprise me. I knew about Twisted’s manhole but yesterday I ran across the code module, which is in the Python standard library.

The code module allows you to create a fully fledged Python REPL. So, in order to try it out I decided to implement a “backdoor” server for Tornado applications.

The idea is pretty simple, use Tornado’s TCPServer to serve a Python interpreter. Of course you should only do this in localhost! The can come really useful for debugging, as it doesn’t take any resources from the application, and allows for live inspection.

The result is here on GitHub.

Here is an example, hope you like it!

[gist]https://gist.github.com/3739834[/gist]

:wq

Integrating Twisted and Tornado with pyuv

Posted on by saghul Leave a comment

With yesterday’s pyuv release the door was open for integrating pyuv with other event loops or applications. Thanks to the Poll handle we can now create a regular socket in Python and put it in pyuv’s event loop, so we can also use pyuv to replace other event loops ;–)

I created a couple of projects for toying around with this feature. The first project implements a Tornado IOLoop which runs on top of pyuv, and the second one implements a Twisted reactor on top of pyuv.

They are not feature complete yet, but basics are working and I’ll be adding more features as time allows.

Go check them out on GitHub!

:wq

A ZeroMQ to WebSocket gateway, take 2

Posted on by saghul 4 Comments

A while ago I posted a way to build a gateway between ZeroMQ and WebSocket. That mechanism required a custom WebSocket server which has to be maintained and I wasn’t really happy with it anyway so I thought I’d find another way.

Moreover, WebSocket is probably not enough. Since the WebSocket specification has changed a lot, different browsers have implemented different versions of the draft. WebSocket could even be disabled god know why, so we need a fallback mechanism. And then I discovered SocketIO. With SocketIO you can establish a persistent connection between a browser and a server in many different ways: WebSocket, flash socket, long polling, etc. This is great stuff.

If I’m not mistaken SocketIO was born for NodeJS but then was ported to some other programming languages and frameworks. The one I’m particularly interested in is TornadIO. TornadIO is a SocketIO implementation on top of Tornado. In case you have lived in a cave for the last several years ;–) you should know that Tornado is the webserver that powered FriendFeed, which was acquired by Facebook.

So, we now have a nice persistent connection between the browser (using SocketIO JavaScript library) and our server (using TornadIO), but how do we add ZeroMQ to the mix? It couldn’t be easier: ZeroMQ added a way to integrate its event loop with the one in Tornado, so there is really nothing else to be done to integrate them. :–)

Here is an example of a ZeroMQ – WebSocket (SocketIO really) gateway built as I described above. The client will connect to the server (using the web browser) through SocketIO and will just sit there waiting to get data. The server will push anything that arrives through the ZeroMQ socket to all clients connected through SocketIO and the messages will just get printed on the screen. It’s a simple example, but you get the idea ;–)

[gist]https://gist.github.com/1222590[/gist]

:wq