Re: What do you think about Zed Shaw / Mongrel2's "SuperPoll"?

Hi Carlo,

On Thu, Aug 04, 2011 at 01:44:10PM -0700, carlo flores wrote:
> Hey Willy, all.
>
> We are playing with Mongrel2 and found this post from Zed interesting and --
> for myself -- surprising: http://sheddingbikes.com/posts/1280829388.html
>
> Have you had a chance to read, think about, or respond to it with regards to
> poll vs epoll in HAProxy versus the number of active and total connections?

No, I didn't know about this article, and I wouldn't trust it too much considering the tests were performed in the context of a server that makes haproxy a success :-)

The measures I've done indicate very different results from what is posted above. The guy seems to completely forget the cost of setting up the polling list, and processing the results :

• with poll(), you have to rebuild the *whole* fd list upon each pass. If you have more than a few thousands fds, you're terribly slow ;
• in epoll(), the cost of adding/removing an FD to the polling list requires a system call. This is expensive. It's not the active vs total ratio which counts here, it's the frequency at which your FDs switch state because buffers are alternatively full and empty.
• saying that the cost of epoll() is O(N) while you can and should limit the number of output response is non-sense.
• each call to the epoll_wait() syscall is more expensive than a call to poll(), which itself is more expensive than a call to select().
• epoll() supports an event-triggered mechanism which saves you from having to add/remove FDs. Haproxy currently doesn't use it because it was not available in the old implementation it started with. Also it requires an fd cache that we now have with sepoll. I'm considering adding it in the future, or possibly changing sepoll to use epoll_et().

I know for sure that epoll wins over poll with large numbers of fds. When some production servers runs at low CPU usage with more than 100k fds, it's a fact that it's not even possible with poll. I've experienced it in the past, and CPU was at 100% well before reaching these loads, simply because the poll list had to be rebuilt tens of thousands of times per second, and all the CPU was spent doing that instead of doing useful work. That's the reason why I think this guy's tests were incorrect.

Epoll is not easy to efficiently deal with, but you have to build your soft around it for it to be very efficient, not to use it as a drop-in for poll(), which I'm sure this guy did. Most likely his tests were done with an epoll loop which scans the whole fd list and performs the epoll_ctl(ADD/REMOVE), which totally voids the point of the O(1) polling.

In short, what I'd recommend is the following :

• if you have very few FDs to poll (say less than 100), use select(), it always wins ;
• if you have less than a few thousands fds to poll, use poll().
• if you have more than a few thousands fds to poll, use epoll.
• if your fds change status very often, use at least a cache and if possible the event-triggered epoll interface.

Note that FreeBSD's kqueue interface is much nicer than the level-triggered epoll interface. However I would consider it on par with the event-triggered one.

Regards,
Willy Received on 2011/08/05 07:46