You're reading from Data Engineering with Python

Product type Book

Published in Oct 2020

Publisher Packt

ISBN-13 9781839214189

Pages 356 pages

Edition 1st Edition

Languages

Python

Concepts

Data Analysis

Author (1):

Paul Crickard

Table of Contents (21) Chapters

Preface

1. Section 1: Building Data Pipelines – Extract Transform, and Load

2. Chapter 1: What is Data Engineering?

3. Chapter 2: Building Our Data Engineering Infrastructure

4. Chapter 3: Reading and Writing Files

5. Chapter 4: Working with Databases

6. Chapter 5: Cleaning, Transforming, and Enriching Data

7. Chapter 6: Building a 311 Data Pipeline

8. Section 2:Deploying Data Pipelines in Production

9. Chapter 7: Features of a Production Pipeline

10. Chapter 8: Version Control with the NiFi Registry

11. Chapter 9: Monitoring Data Pipelines

12. Chapter 10: Deploying Data Pipelines

13. Chapter 11: Building a Production Data Pipeline

14. Section 3:Beyond Batch – Building Real-Time Data Pipelines

15. Chapter 12: Building a Kafka Cluster

16. Chapter 13: Streaming Data with Apache Kafka

17. Chapter 14: Data Processing with Apache Spark

18. Chapter 15: Real-Time Edge Data with MiNiFi, Kafka, and Spark

19. Other Books You May Enjoy

Leave a review - let other readers know what you think

Appendix

Handling common data issues using pandas

Your data may feel special, it is unique, you have created the world's best systems for collecting it, and you have done everything you can to ensure it is clean and accurate. Congratulations! But your data will almost certainly have some problems, and these problems are not special, or unique, and are probably a result of your systems or data entry. The e-scooter dataset is collected using GPS with little to no human input, yet there are end locations that are missing. How is it possible that a scooter was rented, ridden, and stopped, yet the data doesn't know where it stopped? Seems strange, yet here we are. In this section, you will learn how to deal with common data problems using the e-scooter dataset.

Drop rows and columns

Before you modify any fields in your data, you should first decide whether you are going to use all the fields. Looking at the e-scooter data, there is a field named region_id. This field is a code used...