Reader small image

You're reading from  Driving Data Quality with Data Contracts

Product typeBook
Published inJun 2023
PublisherPackt
ISBN-139781837635009
Edition1st Edition
Right arrow
Author (1)
Andrew Jones
Andrew Jones
author image
Andrew Jones

Andrew Jones is a principal engineer at GoCardless, one of Europe's leading Fintech's. He has over 15 years experience in the industry, with the first half primarily as a software engineer, before he moved into the data infrastructure and data engineering space. Joining GoCardless as its first data engineer, he led his team to build their data platform from scratch. After initially following a typical data architecture and getting frustrated with facing the same old challenges he'd faced for years, he started thinking there must be a better way, which led to him coining and defining the ideas around data contracts. Andrew is a regular speaker and writer, and he is passionate about helping organizations get maximum value from data.
Read more about Andrew Jones

Right arrow

Providing the interfaces to the data

In this section, we’ll use the data contract to provision a Google BigQuery table. This will act as the interface to the data, through which the data generators will make their data available to the data consumers. We’ll learn how to use the contract and its schema to dynamically provision and manage those resources, keeping them in sync with the data contract.

This BigQuery table is the first of our contract-driven resources. To create it, we’ll need to convert our data contract to a custom JSON format that defines a BigQuery table and its schema (https://cloud.google.com/bigquery/docs/reference/rest/v2/tables#TableSchema), as highlighted in the following diagram:

Figure 8.2 – Using the data contract to define and create a BigQuery table

Figure 8.2 – Using the data contract to define and create a BigQuery table

We’ll also need a way to send that JSON to the Google Cloud APIs, which will then create the table. To do that, we are going to make use of an...

lock icon
The rest of the page is locked
Previous PageNext Page
You have been reading a chapter from
Driving Data Quality with Data Contracts
Published in: Jun 2023Publisher: PacktISBN-13: 9781837635009

Author (1)

author image
Andrew Jones

Andrew Jones is a principal engineer at GoCardless, one of Europe's leading Fintech's. He has over 15 years experience in the industry, with the first half primarily as a software engineer, before he moved into the data infrastructure and data engineering space. Joining GoCardless as its first data engineer, he led his team to build their data platform from scratch. After initially following a typical data architecture and getting frustrated with facing the same old challenges he'd faced for years, he started thinking there must be a better way, which led to him coining and defining the ideas around data contracts. Andrew is a regular speaker and writer, and he is passionate about helping organizations get maximum value from data.
Read more about Andrew Jones