A while back I learned about concurrent.futures, a new module included in Python 3.2. It allows you to create a pool of processes or threads to which you can distribute work and then wait for it to finish. The API is dead simple and there is a backport of the module for Python 2 here.

Today I was working with a small tool we have at work to help us build Debian packages for a number of distributions, which basically calls debuild a number of times and I’m very happy with the result after using this module. Since the tool is just a helper it used to just build all packages one after another, but as the number of packages grows, doing the operations in a serialized way just takes too long. Since the tasks are CPU bound, threads are of no use here, because of the GIL, so going multi-process can help here.

The idea is simple: create a process pool as big as the number of cores we have available, send the hard work to them and wait until they are all done. Here is an example of such use case:

[gist]https://gist.github.com/3660656[/gist]

As it can be seen, the API is really a pleasure to use and it did solve the problem effectively. :–)

:wq

It’s been a while since I haven’t posted anything around here, so it’s about time :–)

Today I’m releaseing pythonz a Python installation manager, which works for CPython, Stackless and PyPy.

It’s a fork of pythonbrew, with some features removed, some bugfixes and also some new features. The main goal of this project is to provide the a way for installing different Python interpreters. Just that. How to pick one and use it afterwards is out of the scope of the project, for that I recommend virtualenv + virtualenvwrapper.

Lets see it in action!

First lets install pythonz:

curl -kL https://raw.github.com/saghul/pythonz/master/pythonz-install | bash

Once pythonz is installed add the following line to your .bashrc:

[[ -s $HOME/.pythonz/etc/bashrc ]] && source $HOME/.pythonz/etc/bashrc

Also lets source it now in order to start using it:

source ~/.pythonz/etc/bashrc

Lets install some pythons!

pythonz install --type cpython --no-test 2.7.2
pythonz install --type stackless --no-test  2.7.2
pythonz install --type pypy --url https://bitbucket.org/pypy/pypy/downloads/pypy-1.8-osx64.tar.bz2 1.8

With the above commands you’ll get CPython 2.7.2, Stackless 2.7.2 and PyPy 1.8 installed on your system.

Note that PyPy is installed in binary form, so you’ll need to indicate the correct URL, matching the appropriate version for your architecture.

Now lets create some virtualenvs with these Python versions we just installed:

mkvirtualenv -p ~/.pythonz/pythons/CPython-2.7.2/bin/python cpython272
deactivate
mkvirtualenv -p ~/.pythonz/pythons/Stackless-2.7.2/bin/python stackless272
deactivate
mkvirtualenv -p ~/.pythonz/pythons/PyPy-1.8/bin/python pypy18
deactivate

In order to use one of our newly created virtualenvs we can now use workon and deactivate as usual:

saghul@hal:~$ workon pypy18
(pypy18)saghul@hal:~$ python
Python 2.7.2 (0e28b379d8b3, Feb 09 2012, 18:31:14)
[PyPy 1.8.0 with GCC 4.2.1] on darwin
Type "help", "copyright", "credits" or "license" for more information.
And now for something completely different: ``Therefore, specific information,
I was in an ideal context, I had to realize the faith''
>>>>
(pypy18)saghul@hal:~$ deactivate
saghul@hal:~$

Hope you enjoy using it as much as I do :–)

Get the source on GitHUb!

:wq