Coroutines, Self-Referential Structs, and Pinning

In this chapter, we’ll start by improving our coroutines by adding the ability to store variables across state changes. We’ll see how this leads to our coroutines needing to take references to themselves and the issues that arise as a result of that. The reason for dedicating a whole chapter to this topic is that it’s an integral part of getting async/await to work in Rust, and also a topic that is somewhat difficult to get a good understanding of.

The reason for this is that the whole concept of pinning is foreign to many developers and just like the Rust ownership system, it takes some time to get a good and working mental model of it.

Fortunately, the concept of pinning is not that difficult to understand, but how it’s implemented in the language and how it interacts with Rust’s type system is abstract and hard to grasp.

While we won’t cover absolutely everything about pinning in...

Technical requirements

The examples in this chapter will build on the code from the previous chapter, so the requirements are the same. The examples will all be cross-platform and work on all platforms that Rust (https://doc.rust-lang.org/stable/rustc/platform-support.html) and mio (https://github.com/tokio-rs/mio#platforms) support. The only thing you need is Rust installed and this book’s GitHub repository downloaded locally. All the code in this chapter can be found in the ch09 folder.

To follow the examples step by step, you’ll also need corofy installed on your machine. If you didn’t install it in Chapter 7, install it now by going into the ch07/corofy folder in the repository and running the following:

cargo install --force --path .

We’ll also use delayserver in this example, so you need to open a separate terminal, enter the delayserver folder at the root of the repository, and write cargo run so that it’s ready and available for the...

Improving our example 1 – variables

So, let’s recap what we have at this point by continuing where we left off in the previous chapter. We have the following:

• A Future trait
• A coroutine implementation using coroutine/await syntax and a preprocessor
• A reactor based on mio::Poll
• An executor that allows us to spawn as many top-level tasks as we want and schedules the ones that are ready to run
• An HTTP client that only makes HTTP GET requests to our local delayserver instance

It’s not that bad – we might argue that our HTTP client is a little bit limited, but that’s not the focus of this book, so we can live with that. Our coroutine implementation, however, is severely limited. Let’s take a look at how we can make our coroutines slightly more useful.

The biggest downside with our current implementation is that nothing – and I mean nothing – can live across wait points. It makes sense to tackle this...

Improving our example 2 – references

Let’s set everything up for our next version of this example:

• Create a new folder called b-coroutines-references and copy everything from a-coroutines-variables over to it
• You can change the name of the project so that it corresponds with the folder by changing the name attribute in the package section in Cargo.toml, but it’s not something you need to do for the example to work

Note

You can find this example in this book’s GitHub repository in the ch10/b-coroutines-references folder.

This time, we’ll learn how to store references to variables in our coroutines by using the following coroutine/wait example program:

use std::fmt::Write;
coroutine fn async_main() {
    let mut buffer = String::from("\nBUFFER:\n----\n");
    let writer = &mut buffer;
    println!("Program starting");
    ...

Improving our example 3 – this is… not… good…

Pretend you haven’t read this section title and enjoy the fact that our previous example compiled and showed the correct result.

I think our coroutine implementation is so good now that we can look at some optimizations instead. There is one optimization in our executor in particular that I want to do immediately.

Before we get ahead of ourselves, let’s set everything up:

• Create a new folder called c-coroutines-problem and copy everything from b-coroutines-references over to it
• You can change the name of the project so that it corresponds with the folder by changing the name attribute in the package section in Cargo.toml, but it’s not something you need to do for the example to work

Tip

This example is located in this book’s GitHub repository in the ch09/c-coroutines-problem folder.

With that, everything has been set up.

Back to the optimization. You...

Discovering self-referential structs

What happened is that we created a self-referential struct, initialized it so that it took a pointer to itself, and then moved it. Let’s take a closer look:

1. First, we received a future object as an argument to block_on. This is not a problem since the future isn’t self-referential yet, so we can move it around wherever we want to without issues (this is also why moving futures before they’re polled is perfectly fine using proper async/await).
2. Then, we polled the future once. The optimization we did made one essential change. The future was located on the stack (inside the stack frame of our block_on function) when we polled it the first time.
3. When we polled the future the first time, we initialized the variables to their initial state. Our writer variable took a pointer to our buffer variable (stored as a part of our coroutine) and made it self-referential at this point.
4. The first time we polled the future...

Pinning in Rust

The following diagram shows a slightly more complex self-referential struct so that we have something visual to help us understand:

Figure 9.3 – Moving a self-referential struct with three fields

At a very high level, pinning makes it possible to rely on data that has a stable memory address by disallowing any operation that might move it:

Figure 9.4 – Moving a pinned struct

The concept of pinning is pretty simple. The complex part is how it’s implemented in the language and how it’s used.

Pinning in theory

Pinning is a part of Rust’s standard library and consists of two parts: the type, Pin, and the marker-trait, Unpin. Pinning is only a language construct. There is no special kind of location or memory that you move values to so they get pinned. There is no syscall to ask the operating system to ensure a value stays the same place in memory. It’s only a part...

Improving our example 4 – pinning to the rescue

Fortunately, the changes we need to make are small, but before we continue and make the changes, let’s create a new folder and copy everything we had in our previous example over to that folder:

• Copy the entire c-coroutines-problem folder and name the new copy e-coroutines-pin
• Open Cargo.toml and rename the name of the package e-coroutines-pin

Tip

You’ll find the example code we’ll go through here in this book’s GitHub repository under the ch09/e-coroutines-pin folder.

Now that we have a new folder set up, let’s start making the necessary changes. The logical place to start is our Future definition in future.rs.

future.rs

The first thing we’ll do is pull in Pin from the standard library at the very top:

ch09/e-coroutines-pin/src/future.rs

use std::pin::Pin;

The only other change we need to make is in the definition of poll in our Future trait:

fn...

Summary

What a ride, huh? If you’ve got to the end of this chapter, you’ve done a fantastic job, and I have good news for you: you pretty much know everything about how Rust’s futures work and what makes them special already. All the complicated topics are covered.

In the next, and last, chapter, we’ll switch over from our hand-made coroutines to proper async/await. This will seem like a breeze compared to what you’ve gone through so far.

Before we continue, let’s stop for a moment and take a look at what we’ve learned in this chapter.

First, we expanded our coroutine implementation so that we could store variables across wait points. This is pretty important if our coroutine/wait syntax is going to rival regular synchronous code in readability and ergonomics.

After that, we learned how we could store and restore variables that held references, which is just as important as being able to store data.

Next, we saw firsthand...