Measuring the performance of data movement

ADF provides a rich set of performance metrics under its Monitoring tab. In the following example, we have a sample Copy Data activity as part of a pipeline called FetchDataFromBlob, which copies data from Blob storage into Azure Data Lake Storage Gen2 (ADLS Gen2). If you click on the Pipeline runs tab under the Monitoring tab, you will see the details of each of the pipelines. If you click on any of the steps, you will see diagnostic details:

Figure 13.18 – Data movement performance details

This is how you can monitor the performance of data movement. You can learn more about Copy Data monitoring here: https://docs.microsoft.com/en-us/azure/data-factory/copy-activity-monitoring.

Let's next look at how to monitor overall pipeline performance.