Reader small image

You're reading from  Modern Data Architectures with Python

Product typeBook
Published inSep 2023
Reading LevelExpert
PublisherPackt
ISBN-139781801070492
Edition1st Edition
Languages
Concepts
Right arrow
Author (1)
Brian Lipp
Brian Lipp
author image
Brian Lipp

Brian Lipp is a Technology Polyglot, Engineer, and Solution Architect with a wide skillset in many technology domains. His programming background has ranged from R, Python, and Scala, to Go and Rust development. He has worked on Big Data systems, Data Lakes, data warehouses, and backend software engineering. Brian earned a Master of Science, CSIS from Pace University in 2009. He is currently a Sr. Data Engineer working with large Tech firms to build Data Ecosystems.
Read more about Brian Lipp

Right arrow

Terraform setup

We will be using the infrastructure repo to store our infrastructure as code. I will go through the Terraform code in a little bit, but keep in mind we are using pre-commit on this repo also. It will reformat, lint, and syntax check your Terraform code. I can’t stress enough how useful this is, and I wish more teams followed this approach.

Initial file setup

I’m going to shy away from repeating the exact same setup for each Python repo given, in this chapter, we will only have the base template created. Instead, I will walk through one repo and the infrastructure repo and explain the key files.

I create a visible tree of the folder structure I used in Windows. The majority of the work is done in Linux but here and there, I am switching to Windows:

tree /f

Schema repository

Here, we can see the basic folder structure. I have removed anything not committed or not useful to explain. One example might be pytest cache folders. I will show...

lock icon
The rest of the page is locked
Previous PageNext Page
You have been reading a chapter from
Modern Data Architectures with Python
Published in: Sep 2023Publisher: PacktISBN-13: 9781801070492

Author (1)

author image
Brian Lipp

Brian Lipp is a Technology Polyglot, Engineer, and Solution Architect with a wide skillset in many technology domains. His programming background has ranged from R, Python, and Scala, to Go and Rust development. He has worked on Big Data systems, Data Lakes, data warehouses, and backend software engineering. Brian earned a Master of Science, CSIS from Pace University in 2009. He is currently a Sr. Data Engineer working with large Tech firms to build Data Ecosystems.
Read more about Brian Lipp