Fan-Out

The purpose of the Fan-Out design pattern is to divide the workload among multiple concurrent processors, or workers, efficiently. To grasp this concept better, let’s revisit the previous section but consider a specific problem: what if there’s a significant disparity in the amount of work at different stages in our pipeline?

For instance, fetching HTML content might take much longer than parsing it. In such cases, it makes sense to distribute the heavy lifting across multiple coroutines. In the previous example, each channel had only one coroutine reading from it. However, it’s possible for multiple coroutines to consume from a single channel, effectively sharing the workload.

To simplify the problem we’re about to discuss, let’s assume we have only one coroutine producing some results:

fun CoroutineScope.generateWork() = produce {
    for (i in 1..10_000) {
        send("page$i")
    }
    close()
}

And we’...