Using the data factory to manipulate data in the Data Lake
In the previous section, we created the Data Lake Analytics Resource for the U-SQL task:
- Even though possible, it is not at all straightforward to run U-SQL to connect directly to an SQL database. It involves tweaking firewalls and permissions. This is why we do not cover this part in the next section, which describes how to run a U-SQL job directly from the Data Lake Analytics resource.
- It is much simpler to copy data from an SQL Server database to a file on Azure Blob Storage via the Azure Data Factory.
- In this section, we show how to do this and then how to manipulate the copied data with U-SQL using the Azure Data Factory.
We will now create a pipeline in Azure Data Factory that will do the following:
- Task 1: Import data from SQL Server (from a view) into a file on blob storage
- Task 2: Use U-SQL to export summary data to a file on blob storage
Task 1 – copy/import data from SQL Server to a blob storage file using data factory
Let's create...
