A Contract-Driven Data Architecture

In the previous chapter, we saw exactly what makes up a data contract. In this chapter, we’re going to build on that by looking at how we can use the data contract to drive our data architecture. We’ll introduce the concept of a contract-driven data architecture and show how powerful this can be. We believe this is a step-change in how we build data platforms, and we’ll discuss the many benefits we get when adopting this architecture pattern.

As part of that discussion, we’ll introduce the three principles that unlock those benefits: autonomy, guardrails, and consistency, and you’ll learn how those principles benefit the data generators, the data consumers, and the organization. To promote autonomy, we need to provide tooling that can be self-served by the data generators. We’ll finish this chapter by looking at why that is important and show an example of how to achieve it.

By the end of this chapter...

A step-change in building data platforms

To start this section, we’ll explain exactly what we mean by a contract-driven data architecture. We’ll explore how it is powered by using data contracts as the place to capture the metadata that describes the data, and we’ll see just how powerful it can be to create a contract-driven data architecture. We’ll show why we believe it is a step-change in building data platforms.

We’ll finish by walking through a case study from GoCardless, where we implemented a solution we thought was promoting autonomy but wasn’t as successful as we expected! What we learned from that greatly influenced our implementation of data contracts, where we have been much more successful in promoting autonomy through a self-serve interface.

We’ll explore the following topics in turn:

• Building generic data tooling
• Introducing a data infrastructure team
• A case study from GoCardless in promoting autonomy...

Introducing the principles of a contract-driven data architecture

Building a contract-driven data architecture provides many benefits to both the data generators and consumers, and the wider organization. These benefits are achieved through these three principles:

• Automation
• Guidelines and guardrails
• Consistency

Let’s look at each of these in turn.

Automation

There are several common tasks that need to be carried out on the data and the resources we use to manage it, no matter what that data is and who owns it. These tasks are great candidates to automate, reducing the effort the data generators need to spend managing the data.

The resources required for our data will almost always include the tables in the data warehouse. We can use the data contract to automate the creation and management of that table, for example, by creating the table when the contract is created and keeping the schema of the table in sync with the schema in the contract...

Providing self-served data infrastructure

Data generators must be able to create and manage their data products with agility and autonomy if we are going to improve the accessibility of quality data that leads to valuable business outcomes.

To enable that, the tooling implemented as part of our contract-driven architecture needs to be self-servable by those data generators. There should be no waiting on a central data or operations teams for review, slowing the data generators down and becoming a bottleneck.

We can be confident in allowing this because we have implemented the guidelines and guardrails that manage the risks, as we discussed in the previous section. That allows us to trust our data generators, and by showing we trust them we are promoting a sense of ownership of the data. That sense of ownership automatically translates into a feeling of responsibility and accountability for the data, and the data products they are providing.

As we’ve discussed throughout...

Summary

In this chapter, we introduced the concept of a contract-driven data architecture. This is an architecture driven by data contracts and the metadata we define within them. We showed how powerful this idea is, and why we believe it’s a step-change in how we build data platforms.

We use this pattern to build more generic data tooling, where instead of building similar pipelines as point solutions we can build tooling that doesn’t mandate anything about the data and how it is structured if we have enough context about the data, defined as metadata in the data contract. When adopting this pattern, it’s recommended to build a data infrastructure team, whose remit is to build this tooling for the adoption of all data generators, wherever they are in the organization.

To illustrate how this pattern is different from how we built platforms before, we walked through a case study of a previous service we implemented at GoCardless, the Data Platform Gateway...

Further reading

For more information on the topics covered in this chapter, please see the following resources:

• Implementing Data Contracts at GoCardless: https://medium.com/gocardless-tech/implementing-data-contracts-at-gocardless-3b5c49074d13
• 3 Things Our Software Engineers Love About Data Contracts: https://medium.com/gocardless-tech/3-things-our-software-engineers-love-about-data-contracts-3106e1f1602d
• The Data Engineer is dead, long live the (Data) Platform Engineer by Robert Sahlin: https://robertsahlin.substack.com/p/the-data-engineer-is-dead-long-live
• Data-First Stack as an Enabler for Data Products by Animesh Kumar: https://moderndata101.substack.com/p/data-first-stack-as-an-enabler-for
• Building Great Cloud Security Guardrails by Rich Mogull: https://devops.com/building-great-cloud-security-guardrails/
• How We Use Golden Paths to Solve Fragmentation in Our Software Ecosystem by Gary Niemen https://engineering.atspotify.com/2020/08/how-we-use-golden...