Extending Azure Data Factory with Logic Apps and Azure Functions

The Azure ecosystem comprises a variety of different services. Most of them can be integrated with and connected to Azure Data Factory (ADF). In this chapter, we will show you how to harness the power of serverless execution by integrating some of the most commonly used Azure services: Azure Logic Apps and Azure Functions. These recipes will help you understand how Azure services can be useful in designing Extract, Transform, Load (ETL) pipelines.

We will cover the following recipes in this chapter:

• Triggering your data processing with Logic Apps
• Using the Web activity to call an Azure logic app
• Adding flexibility to your pipelines with Azure Functions

Join our book community on Discord

https://packt.link/p3FAF

When your business needs to move data between cloud providers, Azure Data Factory presents a convenient and robust interface for this task. Microsoft provides connectors to integrate the data factory with multiple third-party services, including Amazon Web Services (AWS) and Google Cloud. In this chapter, we will walk though several illustrative examples on migrating data from these two cloud providers. In addition, you will learn how to use Azure Data Factory's Custom Activity to work with providers who are not supported by Microsoft's built-in connectors.In this chapter, we will cover the following recipes:

• Copying data from Amazon S3 to Azure Blob storage -
• Copying large datasets from S3 to ADLS
• Copying data from Google Cloud Storage to Azure Data Lake
• Copying data from Google BigQuery to Azure Data Lake Store
• Migrating data from Google BigQuery to Azure Synapse
• Copying data from Snowflake to Azure Data Lake Store...

Technical requirements

All recipes in this chapter assume that you have a Microsoft Azure account and an instance of a data factory. Refer to Chapter 1, Getting Started with ADF, for instructions on how to set up your Azure account and create a data factory.For the recipes in this chapter, you will need accounts with sufficient permissions on third-party services. For recipes 1 and 2, you will need to set up an account with AWS. For recipes 3, 4, and 5, you will need a Google Cloud account. For recipe 6 you will need Snowflake account.If you do not have accounts already set up with the aforementioned services, you can do this for free:

• Go to https://aws.amazon.com/console/ to sign up for a free AWS account.
• You will need to know this account's access key ID and secret access key. How you get this information will depend on how your account is set up (in other words, is this a root account or an IAM user?). Refer to https://aws.amazon.com/blogs/security/wheres-my-secret-access-key...

Copying data from Amazon S3 to Azure Blob storage

In this recipe, you will learn how to copy data from an AWS S3 bucket to the Azure Blob storage container using a data factory.

Getting ready

This recipe requires you to have an AWS account and an S3 bucket. Refer to the Technical requirements section to find out how to set up a new AWS account if you do not have one. Once you have your AWS account set up, go to https://s3.console.aws.amazon.com/s3/ to create a bucket. Upload the sample CSV files from https://github.com/PacktPublishing/Azure-Data-Factory-Cookbook/tree/master/data to your bucket.

How to do it…

Rather than designing the pipeline in the Author tab, we will use the Copy Data Wizard. The Copy Data Wizard will walk you through pipeline creation step by step, and will create and run the pipeline for you:

1. Go to the home page of Azure Data Factory and select Ingest tile to start the Copy Data Wizard.

   

   Figure 6.1 – Copy Data Wizard interface

2. We need to define...

Copying large datasets from S3 to ADLS

Azure Data Factory can help you move very large datasets into the Azure ecosystem with speed and efficiency. The key to moving large datasets is data partitioning. The way you partition depends heavily on the nature of your data.In the following recipe, we will illustrate a methodology to utilize a data partitioning table for moving a large dataset. We will use a public Common Crawl dataset, which contains petabytes of web crawl data from 2008 to the present day. It is a public dataset hosted on the AWS S3 platform. We will only use a small subset of this data for our example, enough to illustrate the power of data factory parallel processing.

Getting ready

In order to access Amazon Web Services, such as an S3 bucket, you need to have proper credentials. These credentials consist of an access key ID (for example, AKFAGOKFOLNN7EXAMPL8) and the secret access key itself (for example, pUgkrUXtPFEer/PO9rbNG/bPxRgiMYEXAMPLEKEY). In this book, we will...