Reader small image

You're reading from  Modern Data Architectures with Python

Product typeBook
Published inSep 2023
Reading LevelExpert
PublisherPackt
ISBN-139781801070492
Edition1st Edition
Languages
Concepts
Right arrow
Author (1)
Brian Lipp
Brian Lipp
author image
Brian Lipp

Brian Lipp is a Technology Polyglot, Engineer, and Solution Architect with a wide skillset in many technology domains. His programming background has ranged from R, Python, and Scala, to Go and Rust development. He has worked on Big Data systems, Data Lakes, data warehouses, and backend software engineering. Brian earned a Master of Science, CSIS from Pace University in 2009. He is currently a Sr. Data Engineer working with large Tech firms to build Data Ecosystems.
Read more about Brian Lipp

Right arrow

Creating GitHub repos

So, we are going to set up our GitHub infrastructure and use GitHub Actions. First things first, let’s create our repositories. They will have empty README files. I am going to create five repositories, one for infrastructure as code, one for docs, one for an ML application, one for an ETL application, and one to manage DDL:

  • infra: gh repo create infra-project --public --add-readme
  • docs: gh repo create docs-project --public --add-readme
  • ML-Job: gh repo create ML-Jobs-project --public --add-readme
  • ETL-Jobs: gh repo create ETL-Jobs-project --public --add-readme
  • SCHEMA-Jobs: gh repo create SCHEMA-Jobs-project --public --add-readme

Now that every repo is created, let’s introduce a new tool we will use on all the Python repositories. We will use Poetry to manage our projects, which is a very easy-to-use package management system. It will also allow you to deploy Python applications very easily to PyPI. To install Poetry...

lock icon
The rest of the page is locked
Previous PageNext Page
You have been reading a chapter from
Modern Data Architectures with Python
Published in: Sep 2023Publisher: PacktISBN-13: 9781801070492

Author (1)

author image
Brian Lipp

Brian Lipp is a Technology Polyglot, Engineer, and Solution Architect with a wide skillset in many technology domains. His programming background has ranged from R, Python, and Scala, to Go and Rust development. He has worked on Big Data systems, Data Lakes, data warehouses, and backend software engineering. Brian earned a Master of Science, CSIS from Pace University in 2009. He is currently a Sr. Data Engineer working with large Tech firms to build Data Ecosystems.
Read more about Brian Lipp