Optimizing Pipelines for Productivity

In this chapter, you will learn the principles of composability to help you become more productive in building pipelines. You will discover strategies to structure your aggregation pipelines and optimize them for reusability and maintainability, enabling you to build effective pipelines rapidly. This chapter will guide you and help you avoid potential pitfalls, such as needing to revisit and refactor your existing aggregation pipelines whenever your data model evolves.

This chapter covers the following topics:

• Understanding and applying the principles of composability
• Tips and best practices for increasing productivity
• Better alternatives for using a project stage

Embrace composability for increased productivity

An aggregation pipeline is an ordered series of instructions, called stages. The entire output of one stage forms the whole input of the next stage, and so on—without any side effects. Pipelines exhibit high composability, where stages are stateless, self-contained components selected and assembled in various combinations (pipelines) to satisfy specific requirements. This property of aggregation pipelines makes iterative prototyping possible, with straightforward testing after each increment.

With MongoDB aggregations, you can take a complex problem, requiring a complex aggregation pipeline, and break it down into straightforward individual stages, where each step can be developed and tested in isolation. To better comprehend composability, it may be helpful to memorize the following visual model, in Figure 2.1:

Figure 2.1: Aggregation pipeline model

Suppose you have two pipelines with each pipeline...

Better alternatives for a projection stage

The typical method for specifying fields to include or exclude in the MongoDB aggregation framework has been the $project stage. This was the only way to define which fields to keep or omit for many earlier versions of MongoDB. However, $project comes with a few usability challenges:

• $project can be confusing and non-intuitive. You can only choose to include fields or exclude fields in a single stage, but not both. However, there is one exception, where you can exclude the _id field and still define other fields to include. This only applies to the _id field and this ambiguity makes $project unintuitive to apply.
• $project can be verbose and inflexible. If you want to define one new field or revise one field, you will have to name all other fields in the projection to include. If each input record has 100 fields and the pipeline needs to employ a $project stage for the first time, defining these files can become time consuming....

Summary

In this chapter, you learned about the concept of composability, equipping you with the knowledge to create efficient and flexible aggregation pipelines. Throughout the chapter, you discovered practical approaches to optimize the pipelines you build for simplicity, reusability, and maintainability.

In the next chapter, you will learn about considerations for optimizing pipeline performance, understanding explain plans, and applying various recommendations that will enable your aggregation pipelines to execute and complete in the minimum amount of time possible.