You're reading from Azure Data Engineering Cookbook Design and implement batch and streaming analytics using Azure Cloud Services

Product type Paperback

Published in Apr 2021

Publisher Packt

ISBN-13 9781800206557

Length 454 pages

Edition 1st Edition

Languages

Python

Tools

Azure

Concepts

Data Engineering

Authors (2):

Ahmad Osama

Nagaraj Venkatesan

View More author details

Table of Contents (11) Chapters

Preface

1. Chapter 1: Working with Azure Blob Storage

2. Chapter 2: Working with Relational Databases in Azure FREE CHAPTER

3. Chapter 3: Analyzing Data with Azure Synapse Analytics

4. Chapter 4: Control Flow Activities in Azure Data Factory

5. Chapter 5: Control Flow Transformation and the Copy Data Activity in Azure Data Factory

6. Chapter 6: Data Flows in Azure Data Factory

7. Chapter 7: Azure Data Factory Integration Runtime

8. Chapter 8: Deploying Azure Data Factory Pipelines

9. Chapter 9: Batch and Streaming Data Processing with Azure Databricks

10. Other Books You May Enjoy

Implementing control flow activities

In this recipe, we'll learn how to implement common control flow activities such as Get Metadata, Filter, ForEach, and Stored Procedure. Control flows help in building complex iterative and conditional logic to support data ingestion.

In this recipe, we'll implement the following scenario using control flow activities.

We have a data lake directory that contains multiple file formats, such as .txt, .csv, and .tif. We need to copy .txt files from the directory to another directory. We'll also maintain a log of the file copy operation in an Azure SQL Database table.

Getting ready

Before you start, do the following:

Log into Azure from PowerShell. To do this, execute the following command and follow the instructions to log into Azure:
```
Connect-AzAccount
```
Open https://portal.azure.com and log in using your Azure credentials.