libuv internals: the OSX select(2) trick

In case you didn’t know, libuv is an asynchronous platform abstraction library which you should totally check out.

libuv does a lot more than abstract epoll, kequeue and friends, and today we’re going to take a look at one of the many tricks libuv provides.

On OSX libuv uses the kqueue interface for polling sockets. This is currently the most efficient way to do so. For file i/o, however, libuv uses a thread pool, but that’s probably a topic for another post 🙂

Back to kqueue. While it works perfectly fine for sockets, we use it for other types of file descriptors too: pipes (with uv_pipe_t) and ttys (with uv_tty_t). On Unix systems, uv_pipe_t can also be used to open an arbitrary file descriptor and treat it as a libuv stream, trhough uv_pipe_open.

When a tty is opened with uv_tty_init, libuv opens /dev/tty instead (it’s a bit more complicated now, but let’s assume that, for the sake of simplicity) in order to be able to put stdin/out/err in non-blocking mode, without affecting processes which share them. There is a problem, however: file descriptors pointing to /dev/tty don’t work with kqueue. Ouch. You can verify that with this little Python script, which is a port of the test libuv does:

import errno
import os
import select


def test_fd(fd):
    kqueue = select.kqueue()
    kevent = select.kevent(fd, filter=select.KQ_FILTER_READ, flags=select.KQ_EV_ADD | select.KQ_EV_ENABLE)
    events = kqueue.control([kevent], 1, 0.001)
    if not events:
        print "fd workss with kqueue"
        return
    assert len(events) == 1
    event = events[0]
    if (event.flags & select.KQ_EV_ERROR) == 0 or event.data != errno.EINVAL:
        print "fd workss with kqueue"
    else:
        print "fd does NOT work with kqueue"


tty_fd = os.open('/dev/tty', os.O_RDWR)
print "Testing if kqueue works with /dev/tty"
test_fd(tty_fd)

 

So, what do we do now? As it turns out, those file descriptors don’t work with poll(2) either… but they do work with select(2)! This means that while we use kqueue for most file descriptors, we can use select(2) when that doesn’t work.

Enter The OSX select(2) Trick (TM) by our OSX expert resident Fedor Indutny. The trick is to spawn an auxiliary thread which will use select(2) on a file descritor which doesn’t work with kqueue and report POLLIN and POLLOUT events to the loop thread, where the read and write operations are performed. Have a look here for the first implementation.

The avid reader might be wondering: “what if we have more than 1024 file descriptors? select is not going to work!” You’re right! This was a problem, so let’s enter The OSX select(2) Trick II: _DARWIN_UNLIMITED_SELECT.

This little gem hidden in the manual page tells us that if _DARWIN_UNLIMITED_SELECT is defined at compilation time, we are allowed to go beyond the FD_SETSIZE limit! We cannot create the fd_set as usual nor use FD_ZERO, we’ll need to manually allocate it and zero it with memset: 1, 2, 3.

So, there you go, this is how libuv is able to use file descrioptors that don’t work with kqueue seamlessly on OSX.

Liked the article? Want me to write more about libuv internals? Do let me know!