You're reading from Azure Data Engineering Cookbook

Product typeBook

Published inApr 2021

Reading LevelBeginner

PublisherPackt

ISBN-139781800206557

Edition1st Edition

Languages

Python

Tools

Azure Stream Analytics

Concepts

Real Time Streaming

Author (1)

Ahmad Osama

Chapter 7: Azure Data Factory Integration Runtime

The Azure Data Factory Integration Runtime (IR) is the compute infrastructure that is responsible for executing data flows, pipeline activities, data movement, and SQL Server Integration Services (SSIS) packages. There are three types of IR: Azure, self-hosted, and Azure SSIS.

The Azure IR is the default IR that is created whenever a new data factory is created. It can process data flows, data movement, and activities.

A self-hosted IR can be installed on-premises or on a virtual machine running the Windows OS. A self-hosted IR can be used to work with data on-premises or in the cloud. It can be used for data movement and activities.

The Azure SSIS IR is used to lift and shift existing SQL SSIS.

In this chapter, we'll learn how to use a self-hosted IR and Azure SSIS IR through the following recipes:

Configuring a self-hosted IR
Configuring a shared self-hosted IR
Migrating an SSIS package to Azure...

Technical requirements

For this chapter, the following are required:

A Microsoft Azure subscription
PowerShell 7
Microsoft Azure PowerShell

Configuring a self-hosted IR

In this recipe, we'll learn how to configure a self-hosted IR and then use the IR to copy files from on-premises to Azure Storage using the Copy data activity.

Getting ready

To get started, do the following:

Log in to https://portal.azure.com using your Azure credentials.
Open a new PowerShell prompt. Execute the following command to log in to your Azure account from PowerShell:
```
Connect-AzAccount
```
You will need an existing Data Factory account. If you don't have one, create one by executing the ~/azure-data-engineering-cookbook\Chapter04\3_CreatingAzureDataFactory.ps1 PowerShell script.

How to do it…

To configure a self-hosted runtime, follow the given steps:

In the Azure portal, open Data Factory, and then open the Manage tab:
Figure 7.1 – Opening the Manage tab – Data Factory
Select New, and then select Azure, Self-Hosted:
Figure 7.2 – Selecting the Azure, Self-Hosted runtime...

Configuring a shared self-hosted IR

A shared self-hosted runtime, as the name suggests, can be shared among more than one data factory. This helps to use a single self-hosted IR to run multiple pipelines. In this activity, we'll learn how to share a self-hosted IR.

Getting ready

To get started, do the following:

Log in to https://portal.azure.com using your Azure credentials.
Open a new PowerShell prompt. Execute the following command to log in to your Azure account from PowerShell:
```
Connect-AzAccount
```
You will need an existing Data Factory account. If you don't have one, create one by executing the ~/azure-data-engineering-cookbook\Chapter04\3_CreatingAzureDataFactory.ps1 PowerShell script.
You need a self-hosted IR. If you don't have one, follow the previous recipe to create one.

How to do it…

Let's start by creating a new Azure data factory:

Execute the following PowerShell command to create an Azure data factory...

Migrating an SSIS package to Azure Data Factory

SQL SSIS is a widely used on-premises ETL tool. In this recipe, we'll learn how to migrate an existing SSIS package to Azure Data Factory.

We'll do this by configuring an Azure SSIS IR, uploading the SSIS package to Azure SQL Database SSISDB, and then executing the package using the Execute SSIS Package activity.

Getting ready

To get started, do the following:

Log in to https://portal.azure.com using your Azure credentials.
Open a new PowerShell prompt. Execute the following command to log in to your Azure account from PowerShell:
```
Connect-AzAccount
```
You will need an existing Data Factory account. If you don't have one, create one by executing the ~/azure-data-engineering-cookbook\Chapter04\3_CreatingAzureDataFactory.ps1 PowerShell script.
Provision an Azure storage account and upload files to it using ~/Chapter06/2_UploadDatatoAzureStorage.ps1.

How to do it…

We'll start...

Executing an SSIS package with an on-premises data store

We may have SSIS packages accessing an on-premises data source or the destination; for example, we may have files in an on-premises file store to be uploaded to Azure SQL Database, or we may have an on-premises database as the source. In such cases, we require the Azure SSIS IR to connect to the on-premises data store. There are two ways to do that:

Configuring Azure SSIS to connect to on-premises using a Point-to-Site VPN, Site-to-Site VPN, or ExpressRoute. There are three steps to this:
1) Set up a Point-to-Site VPN, Site-to-Site VPN, or ExpressRoute between on-premises and Azure.
2) Create a virtual network and join the Azure SSIS IR with the virtual network.
3) Create a virtual network gateway.
Configure Azure SSIS to use a self-hosted IR as a proxy to connect to on-premises.

In this recipe, we'll explore the second option. To get more information on option 1, you can check out https://docs.microsoft...

The rest of the chapter is locked

You have been reading a chapter from

Azure Data Engineering Cookbook

Published in: Apr 2021Publisher: PacktISBN-13: 9781800206557

A free Packt account unlocks extra newsletters, articles, discounted offers, and much more. Start advancing your knowledge today.

undefined

Unlock this book and the full library FREE for 7 days

Get unlimited access to 7000+ expert-authored eBooks and videos courses covering every tech area you can think of

Start free trial

Renews at $15.99/month. Cancel anytime

Author (1)

Ahmad Osama

Ahmad Osama works for Pitney Bowes Pvt. Ltd. as a technical architect and is a former Microsoft Data Platform MVP. In his day job, he works on developing and maintaining high performant, on-premises and cloud SQL Server OLTP environments as well as deployment and automating tasks using PowerShell. When not working, Ahmad blogs at DataPlatformLabs and can be found glued to his Xbox.
Read more about Ahmad Osama

Other recommended products

Related to this chapter

Azure Data Factory Cookbook

With the help of well-structured and practical recipes, this book will teach you how to integrate data from the cloud and on-premise. You’ll learn how to transform, clean, and consolidate data into a single data platform and get to grips with using ADF as the main ETL and orchestration tool for your data warehouse or data platform project.

BookDec 2020382 pages

Professional Azure SQL Database Administration

As the cloud version of SQL Server, Azure SQL Database differs in key ways when it comes to management, maintenance, and administration. It’s important to know how to administer SQL Database to fully benefit from all of the features and functionality that it provides. This book addresses important aspects of an Azure SQL Database instance such as migration, backup restorations, pricing policies, security, scalability, monitoring, performance optimization, high availability, and disaster recovery. It is a complete guide for database administrators, and ideal for those who are planning to migrate from on premise SQL Server database to an Azure SQL Server database.

BookJul 2018398 pages

Professional Azure SQL Database Administration

This book is your one-stop solution to learning all that is needed to migrate a traditional on-premise SQL server database to a cloud-based solution with Microsoft Azure. Built with database administrators in mind, this book emulates different scenarios you might come across while working with large, complex SQL database migrations and provides solutions for effectively managing the migrated databases.

BookJul 2019562 pages

Limitless Analytics with Azure Synapse

This book helps you understand the basic concepts and techniques of using Azure Synapse step-by-step. You'll gradually gain the skills you need to work with data and develop analytics solutions using the Azure analytics platform even with no prior knowledge of Azure.

BookJun 2021392 pages

Professional Azure SQL Managed Database Administration

Whether it is learning different techniques to monitor and tune an Azure SQL database or improving performance using in-memory technology, this book will enable you to make the most out of Azure SQL database features and functionality for data management solutions.

BookMar 2021724 pages

Learning Microsoft Azure Storage

Microsoft Azure Storage is the bedrock of Microsoft's core storage solution offering in Azure. No matter what solution you are building for the cloud, you'll find a compelling use for Azure Storage. This book will help you get up-to-speed quickly on Microsoft Azure Storage by teaching you how to use the different storage services. You will be able to leverage secure design patterns based on real-world scenarios and develop a strong storage foundation for Azure virtual machines.

BookNov 2017276 pages

Azure Databricks Cookbook

The Azure Databricks Cookbook shows you how to work with the latest as well as older versions of Apache Spark and integrate with various Azure resources for orchestrating, deploying, and monitoring big data solutions. You'll use Azure Databricks to build end-to-end solutions and address challenges in securing, productionizing, and monitoring them.

BookSep 2021452 pages

ETL with Azure Cookbook

This book will take you through hand-on recipes for extracting, transforming, and loading data using big data tools and Azure services such as Data Factory and Azure Databricks. You will learn how to interact effectively with Azure services, along with covering automation with BIML and data profiling in Azure.

BookSep 2020446 pages

Hands-On Data Warehousing with Azure Data Factory

Azure Data Factory (ADF) is a Microsoft Azure PaaS solution which supports data movement between many on premises and cloud data sources. This book covers custom tailored tutorials to help you develop , maintain and troubleshoot data movement processes and environments using Azure Data Factory V2 and SQL Server Integration Services 2017

BookMay 2018284 pages

Cloud Scale Analytics with Azure Data Services

This book will help you to understand the architectural components of a modern data warehouse and select those suitable for your requirements. You’ll learn everything from how to integrate your source data into Azure Data Lake at scale to how to structure your analytical data estate and more.

BookJul 2021520 pages

Professional SQL Server High Availability and Disaster Recovery

The Professional SQL Server High Availability and Disaster Recovery book explains the high availability and the disaster recovery technologies, their technical implementation, and different topologies that you can use when creating a highly available infrastructure with hybrid topologies.

BookJan 2019564 pages

Cloud Analytics with Microsoft Azure

Cloud Analytics with Microsoft Azure enables you to understand the design and business considerations that you must keep in mind while planning to adopt the cloud analytics model for your business.

BookJan 2021184 pages

Personalised recommendations for you

Based on your interests and search pattern

Et al.

Ever wonder why speech recognition systems don't understand the Scottish accent, or what would happen if an astronaut only ate mac 'n' cheese, or other spurious reflections you'd have at a bar? We did, then collated those deliberations into absurd research articles with fake figures and methodologies inspired by even more fictionally absurd studies.

BookAug 2023230 pages5

Generative AI with LangChain

This book is a comprehensive introduction to LLMs and LangChain, demystifying the basic mechanics of LangChain, its functionalities, and the myriad of applications it can be integrated into.

BookDec 2023360 pages4

Generative AI with LangChain

This book is a comprehensive introduction to LLMs and LangChain, demystifying the basic mechanics of LangChain, its functionalities, and the myriad of applications it can be integrated into.

BookDec 2023360 pages5

Generative AI with LangChain

This book is a comprehensive introduction to LLMs and LangChain, demystifying the basic mechanics of LangChain, its functionalities, and the myriad of applications it can be integrated into.

BookDec 2023360 pages1

Generative AI with LangChain

This book is a comprehensive introduction to LLMs and LangChain, demystifying the basic mechanics of LangChain, its functionalities, and the myriad of applications it can be integrated into.

BookDec 2023360 pages5

Mastering Tableau 2023

This book is a comprehensive resource to mastering your Tableau skills and becoming a BI expert. As you progress, you will learn how to build advanced dashboards and improve your storytelling to derive key business insight, as well as make you well-versed with advanced functionalities of Tableau in the business intelligence domain.

BookAug 2023684 pages

Building AI Applications with ChatGPT APIs

This guide covers all ChatGPT API features for effortless creation of robust AI powered apps. With its help, you’ll be able to leverage ChatGPT’s cutting-edge NLP models to take your app development skills to the next level. You’ll also work on ten exciting projects that will give you the practical know-how that you can apply to your existing applications.

BookSep 2023258 pages5

Building AI Applications with ChatGPT APIs

This guide covers all ChatGPT API features for effortless creation of robust AI powered apps. With its help, you’ll be able to leverage ChatGPT’s cutting-edge NLP models to take your app development skills to the next level. You’ll also work on ten exciting projects that will give you the practical know-how that you can apply to your existing applications.

BookSep 2023258 pages2

Data Engineering with AWS

Embark on a journey to master data engineering pipelines on AWS! Our book offers a hands-on experience of AWS services for ingesting, transforming, and consuming data. Whether you're an absolute beginner or someone with basic data engineering experience, this guide is an indispensable resource.

BookOct 2023636 pages5

Modern Data Architecture on AWS

Every organization wants an agile, performant, and cost-effective data platform that meets all their current and future business needs. Purpose-built AWS analytics services and their features play a big part in building such a modern data platform. This book brings to you all the design and architectural patterns that’ll help you achieve this goal.

BookAug 2023420 pages5

Practical Guide to Applied Conformal Prediction in Python

Discover the power of Conformal Prediction with the "Practical Guide to Applied Conformal Prediction in Python." Master the latest techniques to quantify uncertainty in machine learning and computer vision models, and seamlessly apply them to your industry applications.

BookDec 2023240 pages

TinyML Cookbook

With over 70 project-based recipes, the TinyML Cookbook is a practical guide that will help you to get the most out of your microcontrollers. It provides a comprehensive understanding of the theoretical foundations while giving you hands-on experience training ML models for deployment on Arduino Nano 33 BLE Sense, Raspberry Pi Pico, and SparkFun RedBoard Artemis Nano microcontrollers.

BookNov 2023664 pages