Agile Development

In the previous chapter, we completed the basics of coding with dbt, explained how to write macros and tests, and completed an MVP of our target architecture.

In this chapter, you will learn to develop with agility by mixing philosophy and practical hints, see how to keep a backlog agile through the phases of your projects, and take a deep dive into building data marts.

In this chapter, you will learn about the following topics:

• Agile development and collaboration
• Applying agile to data engineering
• Building reports in an agile way

Technical requirements

You can read and understand this chapter without reading the previous ones as long as you know the basics of dbt.

There are no code samples for this chapter.

Agile development and collaboration

In this chapter, we will touch on probably the most important part of the development cycle, the one that contributes the most to project success and delivery time – enabling your people to be productive and eager to work on a project as a team by helping each other.

With over two decades of experience working in different countries, companies, sectors, and functions, I can say that developers love their work, and they hate doing nothing or, even worse, stupid, pointless things.

You can be assured that as long as your developers have a clear picture of what to build and small enough incremental goals to achieve, they will build your project, as this is what brings them satisfaction.

The agile development part of this chapter concentrates on making your people work on what matters most, reducing useless work as much as possible. The collaboration part ensures that they do not work isolated but are aligned, and they help and coach each...

Applying agile to data engineering

Looking at the four statements of the Agile Manifesto, the key message is that people, results, and following real-world concerns are more important than plans and papers. This does not mean that the other elements are bad per se, just that they are not the most important ones, and their importance is in contributing to the other elements.

Here are a few general considerations that apply to the data engineering domain:

• Agile is not improvising. Having a plan is important, as it is the understanding that any plan is not set in stone, and it will keep changing. The point of the plan is to keep track of where we are, the current desired goal, and the knowledge that we have accumulated about the surrounding areas of the domain. Quick failures are important to learn blocked roads as soon as possible .
• Pick a direction in collaboration with the people who know the domain (aka business) and the ones who know the technology. Deliver and verify...

Building reports in an agile way

In this section, we will decompose in stories a feature to build a report in a very general way that you can use for any report and, pretty much, for any other result deliverable in a data mart.

The “F1 – building a customer current position report” feature can be split into stories of these five types:

• S1 – designing a light data model for the data mart to power the F1 report.
• S2 – designing a light data model for the REF layer to power the data mart for the F1 report.
• S3.x – developing with dbt models the pipeline for the DIM_x / FACT_x / REF_x table. This is not one story but a story for each table in the data mart, plus each necessary support table in the REF layer. We will discuss more details later.
• S4 – an acceptance test of the data produced in the data mart.
• S5 – development and verification of the report in the BI application.

These five types of stories...

Summary

In this chapter, you learned some agile philosophy and practical hints, starting from the commencement of your project, managing the backlog, and finally, taking a deep dive into how to organize your work to build reporting data marts. Good job!

In the next chapter, you will learn about collaboration and using Git alongside dbt Cloud to coallborate.