Create Your Own Event Queue

In this chapter, we’ll create a simple version of an event queue using epoll. We’ll take inspiration from mio (https://github.com/tokio-rs/mio), a low-level I/O library written in Rust that underpins much of the Rust async ecosystem. Taking inspiration from mio has the added benefit of making it easier to dive into their code base if you wish to explore how a real production-ready library works.

By the end of this chapter, you should be able to understand the following:

• The difference between blocking and non-blocking I/O
• How to use epoll to make your own event queue
• The source code of cross-platform event queue libraries such as mio
• Why we need an abstraction layer on top of epoll, kqueue, and IOCP if we want a program or library to work across different platforms

We’ve divided the chapter into the following sections:

• Design and introduction to epoll
• The ffi module
• The Poll module
• The...

Technical requirements

This chapter focuses on epoll, which is specific to Linux. Unfortunately, epoll is not part of the Portable Operating System Interface (POSIX) standard, so this example will require you to run Linux and won’t work with macOS, BSD, or Windows operating systems.

If you’re on a machine running Linux, you’re already set and can run the examples without any further steps.

If you’re on Windows, my recommendation is to set up WSL (https://learn.microsoft.com/en-us/windows/wsl/install), if you haven’t already, and install Rust on the Linux operating system running on WSL.

If you’re using Mac, you can create a virtual machine (VM) running Linux, for example, by using the QEMU-based UTM application (https://mac.getutm.app/) or any other solution for managing VMs on a Mac.

A last option is to rent a Linux server (there are even some providers with a free layer), install Rust, and either use an editor such as Vim or Emacs...

Design and introduction to epoll

Okay, so this chapter will be centered around one main example you can find in the repository under ch04/a-epoll. We’ll start by taking a look at how we design our example.

As I mentioned at the start of this chapter, we’ll take our inspiration from mio. This has one big upside and one downside. The upside is that we get a gentle introduction to how mio is designed, making it much easier to dive into that code base if you want to learn more than what we cover in this example. The downside is that we introduce an overly thick abstraction layer over epoll, including some design decisions that are very specific to mio.

I think the upsides outweigh the downsides for the simple reason that if you ever want to implement a production-quality event loop, you’ll probably want to look into the implementations that are already out there, and the same goes for if you want to dig deeper into the building blocks of asynchronous programming...

The ffi module

Let’s start with the modules that don’t depend on any others and work our way from there. The ffi module contains mappings to the syscalls and data structures we need to communicate with the operating system. We’ll also explain how epoll works in detail once we have presented the syscalls.

It’s only a few lines of code, so I’ll place the first part here so it’s easier to keep track of where we are in the file since there’s quite a bit to explain. Open the ffi.rs file and write the following lines of code:

ch04/a-epoll/src/ffi.rs

pub const EPOLL_CTL_ADD: i32 = 1;
pub const EPOLLIN: i32 = 0x1;
pub const EPOLLET: i32 = 1 << 31;
#[link(name = "c")]
extern "C" {
  pub fn epoll_create(size: i32) -> i32;
  pub fn close(fd: i32) -> i32;
  pub fn epoll_ctl(epfd: i32, op: i32, fd: i32, event: *mut Event) -> i32;
  pub fn epoll_wait(epfd: i32, events...

The Poll module

If you haven’t written or copied the code we presented in the Design and introduction to epoll section, it’s time to do it now. We’ll implement all the functions where we just had todo!() earlier.

We start by implementing the methods on our Poll struct. First up is opening the impl Poll block and implementing the new function:

ch04/a-epoll/src/poll.rs

impl Poll {
    pub fn new() -> Result<Self> {
        let res = unsafe { ffi::epoll_create(1) };
        if res < 0 {
            return Err(io::Error::last_os_error());
        }
        Ok(Self {
            registry: Registry { raw_fd: res },
      ...

The main program

Let’s see how it all works in practice. Make sure that delayserver is up and running, because we’ll need it for these examples to work.

The goal is to send a set of requests to delayserver with varying delays and then use epoll to wait for the responses. Therefore, we’ll only use epoll to track read events in this example. The program doesn’t do much more than that for now.

The first thing we do is to make sure our main.rs file is set up correctly:

ch04/a-epoll/src/main.rs

use std::{io::{self, Read, Result, Write}, net::TcpStream};
use ffi::Event;
use poll::Poll;
mod ffi;
mod poll;

We import a few types from our own crate and from the standard library, which we’ll need going forward, as well as declaring our two modules.

We’ll be working directly with TcpStreams in this example, and that means that we’ll have to format the HTTP requests we make to our delayserver ourselves.

The server will accept...

Summary

The concept of epoll, kqueue, and IOCP is pretty simple at a high level, but the devil is in the details. It’s just not that easy to understand and get it working correctly. Even programmers who work on these things will often specialize in one platform (epoll/kqueue or Windows). It’s rare that one person will know all the intricacies of all platforms, and you could probably write a whole book about this subject alone.

If we summarize what you’ve learned and got firsthand experience with in this chapter, the list is quite impressive:

• You learned a lot about how mio is designed, enabling you to go to that repository and know what to look for and how to get started on that code base much easier than before reading this chapter
• You learned a lot about making syscalls on Linux
• You created an epoll instance, registered events with it, and handled those events
• You learned quite a bit about how epoll is designed and its API
• You learned...