Data Engineering with Python

By Paul Crickard
  • Instant online access to over 8,000+ books and videos
  • Constantly updated with 100+ new titles each month
  • Breadth and depth in over 1,000+ technologies
  1. Section 1: Building Data Pipelines – Extract Transform, and Load

About this book

Data engineering provides the foundation for data science and analytics, and forms an important part of all businesses. This book will help you to explore various tools and methods that are used for understanding the data engineering process using Python.

The book will show you how to tackle challenges commonly faced in different aspects of data engineering. You’ll start with an introduction to the basics of data engineering, along with the technologies and frameworks required to build data pipelines to work with large datasets. You’ll learn how to transform and clean data and perform analytics to get the most out of your data. As you advance, you'll discover how to work with big data of varying complexity and production databases, and build data pipelines. Using real-world examples, you’ll build architectures on which you’ll learn how to deploy data pipelines.

By the end of this Python book, you’ll have gained a clear understanding of data modeling techniques, and will be able to confidently build data engineering pipelines for tracking data, running quality checks, and making necessary changes in production.

Publication date:
October 2020
Publisher
Packt
Pages
356
ISBN
9781839214189

 

Section 1: Building Data Pipelines – Extract Transform, and Load

This section will introduce you to the basics of data engineering. In this section, you will learn what data engineering is and how it relates to other similar fields, such as data science. You will cover the basics of working with files and databases in Python and using Apache NiFi. Once you are comfortable with moving data, you will be introduced to the skills required to clean and transform data. The section culminates with the building of a data pipeline to extract 311 data from SeeClickFix, transform it, and load it into another database. Lastly, you will learn the basics of building dashboards with Kibana to visualize the data you have loaded into your database.

This section comprises the following chapters:

About the Author

  • Paul Crickard

    Paul Crickard is the author of Leaflet.js Essentials, a Programmer Analyst and former educational facility planner in Albuquerque, New Mexico. He is one of the first planners to adopt Revit and Building Information Modeling (BIM) in the preparation of master plans for the state. Currently, he has been working on the integration of Revit and BIM in Geographic Information Systems (GIS). With a Master's degree in Political Science and a background in Community, Regional, and Educational Facility Planning, Paul combines rigorous social science theory and techniques to practical planning and architectural projects. He has presented papers at the United States Naval Academy's Foreign Affairs Conference and the American Journalism Historians Association Regional Conference in Salt Lake City. He has given demonstrations on the use of Revit, BIM, and GIS to the Public School Facility Authority in New Mexico and the Albuquerque BIM505 Users Group.

    Browse publications by this author
Book Title
Access this book, plus 8,000 other titles for FREE
Access now