Search icon Close icon
Search icon
Subscription
0
Cart icon
Close icon
You have no products in your basket yet
Arrow left icon
All Products
Best Sellers
New Releases
Books
Videos
Audiobooks
Learning Hub
Newsletters
Free Learning
Arrow right icon
Azure Data Engineering Cookbook

You're reading from  Azure Data Engineering Cookbook

Product type Book
Published in Apr 2021
Publisher Packt
ISBN-13 9781800206557
Pages 454 pages
Edition 1st Edition
Languages
Concepts
Author (1):
Ahmad Osama Ahmad Osama
Profile icon Ahmad Osama
LinkedIn
Ahmad Osama works for Pitney Bowes Pvt. Ltd. as a technical architect and is a former Microsoft Data Platform MVP. In his day job, he works on developing and maintaining high performant, on-premises and cloud SQL Server OLTP environments as well as deployment and automating tasks using PowerShell. When not working, Ahmad blogs at DataPlatformLabs and can be found glued to his Xbox.
See other products by Ahmad Osama

Table of Contents (11) Chapters

Preface 1. Chapter 1: Working with Azure Blob Storage 2. Chapter 2: Working with Relational Databases in Azure 3. Chapter 3: Analyzing Data with Azure Synapse Analytics 4. Chapter 4: Control Flow Activities in Azure Data Factory 5. Chapter 5: Control Flow Transformation and the Copy Data Activity in Azure Data Factory 6. Chapter 6: Data Flows in Azure Data Factory 7. Chapter 7: Azure Data Factory Integration Runtime 8. Chapter 8: Deploying Azure Data Factory Pipelines 9. Chapter 9: Batch and Streaming Data Processing with Azure Databricks 10. Other Books You May Enjoy

Chapter 9: Batch and Streaming Data Processing with Azure Databricks

Databricks is a data engineering product built on top of Apache Spark and provides a unified, cloud optimized platform so that you can perform ETL, machine learning, and AI tasks on a large quantity of data.

Azure Databricks, as its name suggests, is the Databricks integration with Azure, which further provides fully managed Spark clusters, an interactive workspace for data visualization and exploration, Azure Data Factory, integration with data sources such as Azure Blob Storage, Azure Data Lake Storage, Azure Cosmos DB, Azure SQL Data Warehouse, and more.

Azure Databricks can process data from multiple and diverse data sources, such as SQL or NoSQL, structured or unstructured data, and also scale up as many servers as required to cater to any exponential data growth.

In this chapter, we'll cover the following recipes:

  • Configuring the Azure Databricks environment
  • Transforming data using...
lock icon The rest of the chapter is locked
arrow left Previous Chapter
Chapter 1 of 11
Next Chapter arrow right
Register for a free Packt account to unlock a world of extra content!
A free Packt account unlocks extra newsletters, articles, discounted offers, and much more. Start advancing your knowledge today.
Sign up now
Unlock this book and the full library FREE for 7 days
Get unlimited access to 7000+ expert-authored eBooks and videos courses covering every tech area you can think of
Start free trial
Renews at $15.99/month. Cancel anytime}

Authors (1)

Profile icon Ahmad Osama
LinkedIn
Ahmad Osama works for Pitney Bowes Pvt. Ltd. as a technical architect and is a former Microsoft Data Platform MVP. In his day job, he works on developing and maintaining high performant, on-premises and cloud SQL Server OLTP environments as well as deployment and automating tasks using PowerShell. When not working, Ahmad blogs at DataPlatformLabs and can be found glued to his Xbox.
See other products by Ahmad Osama

Other recommended products

Related to this chapter
Left arrow icon
Azure Data Factory Cookbook
Azure Data Factory Cookbook
With the help of well-structured and practical recipes, this book will teach you how to integrate data from the cloud and on-premise. You’ll learn how to transform, clean, and consolidate data into a single data platform and get to grips with using ADF as the main ETL and orchestration tool for your data warehouse or data platform project.
Dec 2020 12 hours 44 minutes
Professional Azure SQL Database Administration
Professional Azure SQL Database Administration
As the cloud version of SQL Server, Azure SQL Database differs in key ways when it comes to management, maintenance, and administration. It’s important to know how to administer SQL Database to fully benefit from all of the features and functionality that it provides. This book addresses important aspects of an Azure SQL Database instance such as migration, backup restorations, pricing policies, security, scalability, monitoring, performance optimization, high availability, and disaster recovery. It is a complete guide for database administrators, and ideal for those who are planning to migrate from on premise SQL Server database to an Azure SQL Server database.
Jul 2018 13 hours 16 minutes
Professional Azure SQL Database Administration
Professional Azure SQL Database Administration
This book is your one-stop solution to learning all that is needed to migrate a traditional on-premise SQL server database to a cloud-based solution with Microsoft Azure. Built with database administrators in mind, this book emulates different scenarios you might come across while working with large, complex SQL database migrations and provides solutions for effectively managing the migrated databases.
Jul 2019 18 hours 44 minutes
Limitless Analytics with Azure Synapse
Limitless Analytics with Azure Synapse
This book helps you understand the basic concepts and techniques of using Azure Synapse step-by-step. You'll gradually gain the skills you need to work with data and develop analytics solutions using the Azure analytics platform even with no prior knowledge of Azure.
Jun 2021 13 hours 4 minutes
Professional Azure SQL Managed Database Administration
Professional Azure SQL Managed Database Administration
Whether it is learning different techniques to monitor and tune an Azure SQL database or improving performance using in-memory technology, this book will enable you to make the most out of Azure SQL database features and functionality for data management solutions.
Mar 2021 24 hours 8 minutes
Learning Microsoft Azure Storage
Learning Microsoft Azure Storage
Microsoft Azure Storage is the bedrock of Microsoft's core storage solution offering in Azure. No matter what solution you are building for the cloud, you'll find a compelling use for Azure Storage. This book will help you get up-to-speed quickly on Microsoft Azure Storage by teaching you how to use the different storage services. You will be able to leverage secure design patterns based on real-world scenarios and develop a strong storage foundation for Azure virtual machines.
Nov 2017 9 hours 12 minutes
Azure Databricks Cookbook
Azure Databricks Cookbook
The Azure Databricks Cookbook shows you how to work with the latest as well as older versions of Apache Spark and integrate with various Azure resources for orchestrating, deploying, and monitoring big data solutions. You'll use Azure Databricks to build end-to-end solutions and address challenges in securing, productionizing, and monitoring them.
Sep 2021 15 hours 4 minutes
ETL with Azure Cookbook
ETL with Azure Cookbook
This book will take you through hand-on recipes for extracting, transforming, and loading data using big data tools and Azure services such as Data Factory and Azure Databricks. You will learn how to interact effectively with Azure services, along with covering automation with BIML and data profiling in Azure.
Sep 2020 14 hours 52 minutes
Hands-On Data Warehousing with Azure Data Factory
Hands-On Data Warehousing with Azure Data Factory
Azure Data Factory (ADF) is a Microsoft Azure PaaS solution which supports data movement between many on premises and cloud data sources. This book covers custom tailored tutorials to help you develop , maintain and troubleshoot data movement processes and environments using Azure Data Factory V2 and SQL Server Integration Services 2017
May 2018 9 hours 28 minutes
Cloud Scale Analytics with Azure Data Services
Cloud Scale Analytics with Azure Data Services
This book will help you to understand the architectural components of a modern data warehouse and select those suitable for your requirements. You’ll learn everything from how to integrate your source data into Azure Data Lake at scale to how to structure your analytical data estate and more.
Jul 2021 17 hours 20 minutes
Professional SQL Server High Availability and Disaster Recovery
Professional SQL Server High Availability and Disaster Recovery
The Professional SQL Server High Availability and Disaster Recovery book explains the high availability and the disaster recovery technologies, their technical implementation, and different topologies that you can use when creating a highly available infrastructure with hybrid topologies.
Jan 2019 18 hours 48 minutes
Cloud Analytics with Microsoft Azure
Cloud Analytics with Microsoft Azure
Cloud Analytics with Microsoft Azure enables you to understand the design and business considerations that you must keep in mind while planning to adopt the cloud analytics model for your business.
Jan 2021 6 hours 8 minutes
Right arrow icon

Personalised recommendations for you

Based on your interests and search pattern
Left arrow icon
Et al.
Et al.
Ever wonder why speech recognition systems don't understand the Scottish accent, or what would happen if an astronaut only ate mac 'n' cheese, or other spurious reflections you'd have at a bar? We did, then collated those deliberations into absurd research articles with fake figures and methodologies inspired by even more fictionally absurd studies.
Aug 2023 7 hours 40 minutes
Generative AI with LangChain
Generative AI with LangChain
This book is a comprehensive introduction to LLMs and LangChain, demystifying the basic mechanics of LangChain, its functionalities, and the myriad of applications it can be integrated into.
Dec 2023 12 hours 0 minutes
Generative AI with LangChain
Generative AI with LangChain
This book is a comprehensive introduction to LLMs and LangChain, demystifying the basic mechanics of LangChain, its functionalities, and the myriad of applications it can be integrated into.
Dec 2023 12 hours 0 minutes
Generative AI with LangChain
Generative AI with LangChain
This book is a comprehensive introduction to LLMs and LangChain, demystifying the basic mechanics of LangChain, its functionalities, and the myriad of applications it can be integrated into.
Dec 2023 12 hours 0 minutes
Generative AI with LangChain
Generative AI with LangChain
This book is a comprehensive introduction to LLMs and LangChain, demystifying the basic mechanics of LangChain, its functionalities, and the myriad of applications it can be integrated into.
Dec 2023 12 hours 0 minutes
Mastering Tableau 2023
Mastering Tableau 2023
This book is a comprehensive resource to mastering your Tableau skills and becoming a BI expert. As you progress, you will learn how to build advanced dashboards and improve your storytelling to derive key business insight, as well as make you well-versed with advanced functionalities of Tableau in the business intelligence domain.
Aug 2023 22 hours 48 minutes
Building AI Applications with ChatGPT APIs
Building AI Applications with ChatGPT APIs
This guide covers all ChatGPT API features for effortless creation of robust AI powered apps. With its help, you’ll be able to leverage ChatGPT’s cutting-edge NLP models to take your app development skills to the next level. You’ll also work on ten exciting projects that will give you the practical know-how that you can apply to your existing applications.
Sep 2023 8 hours 36 minutes
Building AI Applications with ChatGPT APIs
Building AI Applications with ChatGPT APIs
This guide covers all ChatGPT API features for effortless creation of robust AI powered apps. With its help, you’ll be able to leverage ChatGPT’s cutting-edge NLP models to take your app development skills to the next level. You’ll also work on ten exciting projects that will give you the practical know-how that you can apply to your existing applications.
Sep 2023 8 hours 36 minutes
Data Engineering with AWS
Data Engineering with AWS
Embark on a journey to master data engineering pipelines on AWS! Our book offers a hands-on experience of AWS services for ingesting, transforming, and consuming data. Whether you're an absolute beginner or someone with basic data engineering experience, this guide is an indispensable resource.
Oct 2023 21 hours 12 minutes
Modern Data Architecture on AWS
Modern Data Architecture on AWS
Every organization wants an agile, performant, and cost-effective data platform that meets all their current and future business needs. Purpose-built AWS analytics services and their features play a big part in building such a modern data platform. This book brings to you all the design and architectural patterns that’ll help you achieve this goal.
Aug 2023 14 hours 0 minutes
Practical Guide to Applied Conformal Prediction in Python
Practical Guide to Applied Conformal Prediction in Python
Discover the power of Conformal Prediction with the "Practical Guide to Applied Conformal Prediction in Python." Master the latest techniques to quantify uncertainty in machine learning and computer vision models, and seamlessly apply them to your industry applications.
Dec 2023 8 hours 0 minutes
TinyML Cookbook
TinyML Cookbook
With over 70 project-based recipes, the TinyML Cookbook is a practical guide that will help you to get the most out of your microcontrollers. It provides a comprehensive understanding of the theoretical foundations while giving you hands-on experience training ML models for deployment on Arduino Nano 33 BLE Sense, Raspberry Pi Pico, and SparkFun RedBoard Artemis Nano microcontrollers.
Nov 2023 22 hours 8 minutes
Right arrow icon