Mostly cat /dev/urandom
Some weeks ago I was searching Twitter for “pyuv” and found a link to an article on pyuv, which was written in Japanese. I contacted the author, @Masahito and he was so kind he translated the post to English. Thanks! You can have a look at it here.
I was happily surprised to see that the perfornace is as good as Tornado’s :–)
It’s been a while since I released pyuv. In this time the libuv guys have been really busy adding lots of cool stuff and with this release pyuv now wraps most parts of libuv. Here is a short changelog highlighting the important additions:
For the next release I plan to add the missing functions for asynchronous filesystem operations and work on Windows support (I even got a pull request to start with, yay!). Stay tuned!
Check it out! https://github.com/saghul/pyuv
Documentation: pyuv.rtfd.org
:wq
After working on this on-and-of on my free time I’m happy to share it. pyuv is a Python interface to libuv, a high-performance, portable library for asynchronous network communications and more. libuv is the platform layer for the well known NodeJS.
pyuv implements the following features:
libuv implements more features, which I plan to implement as time allows. In the meanwhile grab the source and have a look at the examples on the GitHub repository.
Documentation is available through ReadTheDocs here.
Hope you like it! :–)
:wq
A while ago I posted a way to build a gateway between ZeroMQ and WebSocket. That mechanism required a custom WebSocket server which has to be maintained and I wasn’t really happy with it anyway so I thought I’d find another way.
Moreover, WebSocket is probably not enough. Since the WebSocket specification has changed a lot, different browsers have implemented different versions of the draft. WebSocket could even be disabled god know why, so we need a fallback mechanism. And then I discovered SocketIO. With SocketIO you can establish a persistent connection between a browser and a server in many different ways: WebSocket, flash socket, long polling, etc. This is great stuff.
If I’m not mistaken SocketIO was born for NodeJS but then was ported to some other programming languages and frameworks. The one I’m particularly interested in is TornadIO. TornadIO is a SocketIO implementation on top of Tornado. In case you have lived in a cave for the last several years ;–) you should know that Tornado is the webserver that powered FriendFeed, which was acquired by Facebook.
So, we now have a nice persistent connection between the browser (using SocketIO JavaScript library) and our server (using TornadIO), but how do we add ZeroMQ to the mix? It couldn’t be easier: ZeroMQ added a way to integrate its event loop with the one in Tornado, so there is really nothing else to be done to integrate them. :–)
Here is an example of a ZeroMQ – WebSocket (SocketIO really) gateway built as I described above. The client will connect to the server (using the web browser) through SocketIO and will just sit there waiting to get data. The server will push anything that arrives through the ZeroMQ socket to all clients connected through SocketIO and the messages will just get printed on the screen. It’s a simple example, but you get the idea ;–)
[gist]https://gist.github.com/1222590[/gist]
:wq
Today I found myself thinking about a straightforward way of having “thread-local singletons”, that is, only one instance of a given class per thread. I don’t have a clear usercase for this, but I think it could be applied to something like a “one event loop per thread” concept, for example.
This can be done easily by saving an instance in a module level variable stored in a threading.local container, but I wanted to explicitly try to create the object and always get the same instance back, assuming I’m on the same thread.
Here is the solution I came up with:
[gist]https://gist.github.com/1184190[/gist]
A metaclass is used to either create the new class instance or return the existing one from the thread-local storage. Since the class will only be instantiated once (the first time) I have chosen to ignore all initialization attributes, otherwise code would look very weird, because regardless of the initialization attributes, the same object would be returned. This could be improved by keeping a mapping between initialization attributes tuples and object instances, though.
Anyhow, this was a 5 minues quick-and-dirty hack, but do let me know if you hate it a lot :–)
:wq
Yesterday while going through some code with a coworker, we lost 5 minutes trying to find why 2 objects that looked the same where actually not evaluated equal. We did have the __eq__ method defined on the class, yet the comparison was evaluating to False. Why?
Actually we were not testing for equality but for difference, so I thought: “hum, is ne required in this case?”. Actually it is. I felt stupid for not remembering it, but fortunately the Python documentation contained a great explanation:
There are no implied relationships among the comparison operators. The truth of x==y does not imply that x!=y is false. Accordingly, when defining
__eq__(), one should also define
__ne__()so that the operators will behave as expected.
Lets see a quick example showing the failure, and hopefully remember it for the next time:
[gist]https://gist.github.com/1114589[/gist]
:wq
In the last post we saw a simple PubSub application using ZeroMQ. Today we are going to extend that and publish all the data we get from the ZeroMQ publishers to web clients by using WebSocket.
First problem I ran into was choosing a WebSocket server to suit my needs. I had a look at Eventlet, Gevent, txWebsocket and some others, but since what I wanted was quite simple I decided to go and write my own, in order to avoid such dependencies. Communicating both worlds (ZeroMQ and WebSocket) seemed like the big problem, so I thought the best way to solve it would be to put all sockets in a polling loop running on its own thread and avoid possible multithreading issues. This works thanks to the ZeroMQ Poller, which is able to poll ZeroMQ sockets and anything which defines a fileno() method. Awesome stuff.
Since the single threaded polling experiment worked I decided to make a small standalone project (without ZeroMQ, using select.poll instead) with it for possible future stuff. You can find it here.
Back to our gateway, lets go and check the code:
producer:
[gist]https://gist.github.com/1046944[/gist]
As you may have guessed, its the same simple multicast producer we saw on the previous post.
client web page:
[gist]https://gist.github.com/1051836[/gist]
consumer and WebSocket server:
[gist]https://gist.github.com/1051872[/gist]
I had never used poll nor ZeroMQ Poller before, but they were both pretty easy to get started with. In a nutshell, the server will take every piece of data it gets from the ZeroMQ socket and broadcast it to all the connected web clients through web sockets. The gateway is one way only, the input coming from the web sockets is not sent anywhere, I’ll leave that as an experiment for the reader. :–)
To test it just run the server and serve the page. You can use this to serve the page.
:wq
A while back I came across this awesome post by Nicholas Piël about ZeroMQ and I added test ZeroMQ to my never ending ToCheck list. Today I found the excuse to check it out and here is what I did with it.
I wanted to implement a producer-consumer application to aggregate data contained in an arbitrary number of files in an arbitrary number of nodes. ZeroMQ seemed like a good tool for this, so lets see a helloworld example: producer:
[gist]https://gist.github.com/1046936[/gist]
consumer:
[gist]https://gist.github.com/1046938[/gist]
As you can see the code is very straightforward. We create a socket, declare it as a publisher (PUB) or subscriber (SUB) accordingly and subscribe to the topics which we care about (in the case of the subscriber).
Looks like we are writing or reading to/from a standard socket, but this socket is on steroids. You can write to it even if the other end is not connected, all the queuing logic has been taken care for you, it accepts multiple connections, … and many many other things.
UPDATE: After re-reading the docs it seems that ZeroMQ sockets are not really thread-safe unless you pass them using a “full memory barrier”. I guess the fact that Python has the GIL makes my example not crash :–)
ZeroMQ is also thread-safe, and this makes our code a lot simpler than it would if ZeroMQ weren’t thread safe. Lets see an example with 10 threads simultaneously sending messages through a ZeroMQ socket without any locking mechanism.
[gist]https://gist.github.com/1046942[/gist]
The approach we took is very flexible in the sense that any number of producers may connect to our consumer, but they need to know where (in terms of IP and port) our consumer is or will be. Could we change that somehow?
Yes we can. We can use multicast for this. We can make all producers join a multicast group and send their messages there. Then the consumer would also join the same group and get them. Of course this only makes sense in a controlled environment where data is not critical or no non-authorized party is able to join the multicast group.
The funny thing is that we only need to change one line of code in the producer and another one in the consumer in order to make them work in multicast mode.
producer:
[gist]https://gist.github.com/1046944[/gist]
consumer:
[gist]https://gist.github.com/1046947[/gist]
I’ve started a small project I’ll be working on in my free time, hope to post something about it soon :–)
:wq
In case you didn’t know, Tropo is a set simple and powerful APIs that allow you to build rich voice, IM and text applications in the cloud. It supports many programming languages, Python being among them, of course. Development will also be free, so go and get an account to start playing with it!
With Tropo you can develop applications in 2 ways: hosted scripting or web API. In case you are using the web API, the code runs on your server, so there should be no problem there. When using hosted scripting applications, however, we face limitations imposed by the runtime. The Python interpreter is actually Jython 2.5 and we need to be aware of that. One of the things I really miss is some module to encode / decode JSON.
But hey, didn’t we say we could import modules directly from the web in the previous post? Lets just leverage that and build a test application.
My test application will just reply with a random Chuch Norris quote by using the JSON API provided by The Internet Chuck Norris Database. The good thing about Tropo is that you only need to write your application once, but it’s multi-channel: you may connect to it using Jabber and you’ll get a text back, or you may call it using SIP or Skype, and you’ll listen to the quote using text to speech. Cool uh?!
Lets see the code:
[gist]https://gist.github.com/1045544[/gist]
Code should be simple and self-explanatory, leaving out the importing magic. The application is running, you may test it:
Skype Voice: +990009369996104671
SIP Voice: sip:9996104671@sip.tropo.com
Jabber: saghul_test2@tropo.im
Happy Tropo-ing!
:wq
Last week I came across this piece of code (sorry, I lost the original source, if you know it please let me know!) which uses GitHub raw mode and Python’s import hooks to import modules directly from GitHub.
I modified it slightly, to make it a bit more generic and allow you to import Python modules from arbitrary web URLs. Why would you want to do this? Is it safe? Well, you shouldn’t use this unless you are running your script in some restricted environment where you can’t import arbitrary modules. Security could be an issue here, so be careful!
Here is the code:
[gist]https://gist.github.com/1037345[/gist]
It contains 3 test cases that show how it could be used. I’ll post a ‘real world’ usage later today.
:wq