Search icon Close icon
Search icon
Subscription
0
Cart icon
Close icon
You have no products in your basket yet
Save more on your purchases!
Savings automatically calculated. No voucher code required
Arrow left icon
All Products
Best Sellers
New Releases
Books
Videos
Audiobooks
Learning Hub
Newsletters
Free Learning
Arrow right icon
Building ETL Pipelines with Python

You're reading from  Building ETL Pipelines with Python

Product type Book
Published in Sep 2023
Publisher Packt
ISBN-13 9781804615256
Pages 246 pages
Edition 1st Edition
Languages
Concepts
Authors (2):
Brij Kishore Pandey Brij Kishore Pandey
Profile icon Brij Kishore Pandey
LinkedIn
Brij Kishore Pandey stands as a testament to dedication, innovation, and mastery in the vast domains of software engineering, data engineering, machine learning, and architectural design. His illustrious career, spanning over 14 years, has seen him wear multiple hats, transitioning seamlessly between roles and consistently pushing the boundaries of technological advancement. He has a degree in electrical and electronics engineering. His work history includes the likes of JP Morgan Chase, American Express, 3M Company, Alaska Airlines, and Cigna Healthcare. He is currently working as a principal software engineer at Automatic Data Processing Inc. (ADP). Originally from India, he resides in Parsippany, New Jersey, with his wife and daughter.
See other products by Brij Kishore Pandey
Emily Ro Schoof Emily Ro Schoof
Profile icon Emily Ro Schoof
LinkedIn
Emily Ro Schoof is a dedicated data specialist with a global perspective, showcasing her expertise as a data scientist and data engineer on both national and international platforms. Drawing from a background rooted in healthcare and experimental design, she brings a unique perspective of expertise to her data analytic roles. Emily's multifaceted career ranges from working with UNICEF to design automated forecasting algorithms to identify conflict anomalies using near real-time media monitoring to serving as a subject matter expert for General Assembly's Data Engineering course content and design. Her mission is to empower individuals to leverage data for positive impact. Emily holds the strong belief that providing easy access to resources that merge theory and real-world applications is the essential first step in this process.
See other products by Emily Ro Schoof
View More author details

Table of Contents (22) Chapters

Preface 1. Part 1:Introduction to ETL, Data Pipelines, and Design Principles
2. Chapter 1: A Primer on Python and the Development Environment 3. Chapter 2: Understanding the ETL Process and Data Pipelines 4. Chapter 3: Design Principles for Creating Scalable and Resilient Pipelines 5. Part 2:Designing ETL Pipelines with Python
6. Chapter 4: Sourcing Insightful Data and Data Extraction Strategies 7. Chapter 5: Data Cleansing and Transformation 8. Chapter 6: Loading Transformed Data 9. Chapter 7: Tutorial – Building an End-to-End ETL Pipeline in Python 10. Chapter 8: Powerful ETL Libraries and Tools in Python 11. Part 3:Creating ETL Pipelines in AWS
12. Chapter 9: A Primer on AWS Tools for ETL Processes 13. Chapter 10: Tutorial – Creating an ETL Pipeline in AWS 14. Chapter 11: Building Robust Deployment Pipelines in AWS 15. Part 4:Automating and Scaling ETL Pipelines
16. Chapter 12: Orchestration and Scaling in ETL Pipelines 17. Chapter 13: Testing Strategies for ETL Pipelines 18. Chapter 14: Best Practices for ETL Pipelines 19. Chapter 15: Use Cases and Further Reading 20. Index 21. Other Books You May Enjoy

Summary

Within this chapter, we spent time reviewing some of the most popular resources within Amazon’s cloud-based environment. With your local environment now configured to seamlessly integrate with AWS tools, you can harness the power of AWS cloud resources to enhance the scalability and resilience of your pipelines, even at their most basic level. This seamless integration opens up possibilities for adapting and optimizing your pipelines to leverage the robustness and scalability offered by AWS resources. We highly encourage you to spend a couple of hours continuing to explore some of the free offerings within AWS, as well as play around with solely relying on the CLI to get more comfortable with working with AWS from your local device.

Echoing our previous words of caution regarding the pros and cons of a scaling, pay-as-you-go price point, it can be just as dangerous as it is useful; keep in mind that you might shell out more money than you bargained for if you’...

lock icon The rest of the chapter is locked
arrow left Previous Chapter
Chapter 1 of 22
Next Chapter arrow right
Register for a free Packt account to unlock a world of extra content!
A free Packt account unlocks extra newsletters, articles, discounted offers, and much more. Start advancing your knowledge today.
Sign up now
Unlock this book and the full library FREE for 7 days
Get unlimited access to 7000+ expert-authored eBooks and videos courses covering every tech area you can think of
Start free trial
Renews at $15.99/month. Cancel anytime}

Authors (2)

Left arrow icon
Profile icon Brij Kishore Pandey
LinkedIn
Brij Kishore Pandey stands as a testament to dedication, innovation, and mastery in the vast domains of software engineering, data engineering, machine learning, and architectural design. His illustrious career, spanning over 14 years, has seen him wear multiple hats, transitioning seamlessly between roles and consistently pushing the boundaries of technological advancement. He has a degree in electrical and electronics engineering. His work history includes the likes of JP Morgan Chase, American Express, 3M Company, Alaska Airlines, and Cigna Healthcare. He is currently working as a principal software engineer at Automatic Data Processing Inc. (ADP). Originally from India, he resides in Parsippany, New Jersey, with his wife and daughter.
Read more
See other products by Brij Kishore Pandey
Profile icon Emily Ro Schoof
LinkedIn
Emily Ro Schoof is a dedicated data specialist with a global perspective, showcasing her expertise as a data scientist and data engineer on both national and international platforms. Drawing from a background rooted in healthcare and experimental design, she brings a unique perspective of expertise to her data analytic roles. Emily's multifaceted career ranges from working with UNICEF to design automated forecasting algorithms to identify conflict anomalies using near real-time media monitoring to serving as a subject matter expert for General Assembly's Data Engineering course content and design. Her mission is to empower individuals to leverage data for positive impact. Emily holds the strong belief that providing easy access to resources that merge theory and real-world applications is the essential first step in this process.
Read more
See other products by Emily Ro Schoof
Right arrow icon

Personalised recommendations for you

Based on your interests and search pattern
Left arrow icon
Et al.
Et al.
Ever wonder why speech recognition systems don't understand the Scottish accent, or what would happen if an astronaut only ate mac 'n' cheese, or other spurious reflections you'd have at a bar? We did, then collated those deliberations into absurd research articles with fake figures and methodologies inspired by even more fictionally absurd studies.
Aug 2023 7 hours 40 minutes
Generative AI with LangChain
Generative AI with LangChain
This book is a comprehensive introduction to LLMs and LangChain, demystifying the basic mechanics of LangChain, its functionalities, and the myriad of applications it can be integrated into.
Dec 2023 12 hours 0 minutes
Generative AI with LangChain
Generative AI with LangChain
This book is a comprehensive introduction to LLMs and LangChain, demystifying the basic mechanics of LangChain, its functionalities, and the myriad of applications it can be integrated into.
Dec 2023 12 hours 0 minutes
Generative AI with LangChain
Generative AI with LangChain
This book is a comprehensive introduction to LLMs and LangChain, demystifying the basic mechanics of LangChain, its functionalities, and the myriad of applications it can be integrated into.
Dec 2023 12 hours 0 minutes
Generative AI with LangChain
Generative AI with LangChain
This book is a comprehensive introduction to LLMs and LangChain, demystifying the basic mechanics of LangChain, its functionalities, and the myriad of applications it can be integrated into.
Dec 2023 12 hours 0 minutes
Mastering Tableau 2023
Mastering Tableau 2023
This book is a comprehensive resource to mastering your Tableau skills and becoming a BI expert. As you progress, you will learn how to build advanced dashboards and improve your storytelling to derive key business insight, as well as make you well-versed with advanced functionalities of Tableau in the business intelligence domain.
Aug 2023 22 hours 48 minutes
Building AI Applications with ChatGPT APIs
Building AI Applications with ChatGPT APIs
This guide covers all ChatGPT API features for effortless creation of robust AI powered apps. With its help, you’ll be able to leverage ChatGPT’s cutting-edge NLP models to take your app development skills to the next level. You’ll also work on ten exciting projects that will give you the practical know-how that you can apply to your existing applications.
Sep 2023 8 hours 36 minutes
Building AI Applications with ChatGPT APIs
Building AI Applications with ChatGPT APIs
This guide covers all ChatGPT API features for effortless creation of robust AI powered apps. With its help, you’ll be able to leverage ChatGPT’s cutting-edge NLP models to take your app development skills to the next level. You’ll also work on ten exciting projects that will give you the practical know-how that you can apply to your existing applications.
Sep 2023 8 hours 36 minutes
Data Engineering with AWS
Data Engineering with AWS
Embark on a journey to master data engineering pipelines on AWS! Our book offers a hands-on experience of AWS services for ingesting, transforming, and consuming data. Whether you're an absolute beginner or someone with basic data engineering experience, this guide is an indispensable resource.
Oct 2023 21 hours 12 minutes
Modern Data Architecture on AWS
Modern Data Architecture on AWS
Every organization wants an agile, performant, and cost-effective data platform that meets all their current and future business needs. Purpose-built AWS analytics services and their features play a big part in building such a modern data platform. This book brings to you all the design and architectural patterns that’ll help you achieve this goal.
Aug 2023 14 hours 0 minutes
Practical Guide to Applied Conformal Prediction in Python
Practical Guide to Applied Conformal Prediction in Python
Discover the power of Conformal Prediction with the "Practical Guide to Applied Conformal Prediction in Python." Master the latest techniques to quantify uncertainty in machine learning and computer vision models, and seamlessly apply them to your industry applications.
Dec 2023 8 hours 0 minutes
TinyML Cookbook
TinyML Cookbook
With over 70 project-based recipes, the TinyML Cookbook is a practical guide that will help you to get the most out of your microcontrollers. It provides a comprehensive understanding of the theoretical foundations while giving you hands-on experience training ML models for deployment on Arduino Nano 33 BLE Sense, Raspberry Pi Pico, and SparkFun RedBoard Artemis Nano microcontrollers.
Nov 2023 22 hours 8 minutes
Right arrow icon