Microsoft Fabric and Power BI, Azure ML, and Cognitive Services

One of the benefits of Azure Cloud Services is the opportunity to leverage the latest technological innovations without investing additional resources. In this chapter, we delve into several services that provide access to state-of-the-art technologies. Azure Machine Learning (ML) equips data engineers and data scientists with the tools necessary to create, deploy, and manage complex data models. Azure Cognitive Services offers a collection of APIs that grant access to a range of artificial intelligence algorithms (various image classification, natural language processing, and speech recognition algorithms to name just a few areas), which are tailored for business use. Finally, in May 2023, Microsoft unveiled Fabric, a comprehensive integrated analytics solution. Fabric contains a suite of services for data storage, data movement, and data analytics.

Azure Data Factory (ADF) serves as a bridge for integration with...

Technical requirements

For this chapter, you will need the following:

• For the first three recipes, you will need a Power BI or a Fabric subscription. If you do not have one, you can activate a free trial subscription by following these steps:
  1. Navigate to https://app.fabric.microsoft.com/home and sign up for a free Power BI license.

     Note

     You need a business email address to do this.

  2. Once on the Fabric portal, go to the account manager in the top-right corner and click on the Start Trial button in the popup. Your trial should be activated after this, and your account manager should look similar to the following snapshot:

     

     Figure 8.1: Activated Fabric trial

     If you do not see the Start Trial button, you need to work with your Power BI tenant administrator to enable user access to Fabric. Instructions for administrators can be found on the Microsoft site at https://learn.microsoft.com/en-us...

Introducing Microsoft Fabric and Data Factory

Microsoft unveiled its unified business analytics platform Fabric in Spring 2023. Fabric combines the capabilities of Power BI, Azure Synapse, and ADF in one integrated environment.

In this recipe, our goal is to get acquainted with the Microsoft Fabric environment, where we will create the infrastructure and data store necessary for working in Fabric, and load some data from remote storage into the newly created data store.

Getting ready

For this recipe, you need an active Power BI account and Microsoft Fabric activated. Follow the steps described in the Technical requirements section of this chapter to activate the Fabric trial license if you have not done so already.

How to do it...

We shall start by creating a Microsoft Fabric workspace, then we’ll instantiate a Lakehouse and load data using the Microsoft Fabric Copy data wizard:

1. Open Fabric at https://app.fabric.microsoft.com/home and log in to...

Microsoft Fabric Data Factory: A closer look at the pipelines

Microsoft Fabric Data Factory pipelines have many similarities with ADF pipelines. Many activities that are available in ADF (such as control flow and move and transform activities) are also available in Fabric Data Factory, and as time goes by more and more activities (such as data transformation activities) will be available on the Fabric platform.

In this recipe, we shall build a more complex pipeline to parse and load data from a file into a table in the lakehouse storage.

Getting ready

For this recipe, you need an active Power BI account and Microsoft Fabric activated. Follow the steps described in the Technical requirements section of this chapter to activate a Fabric trial license if you have not done so already.

This recipe builds upon the pipeline that we built in the previous recipe. If you do not have the pipeline or did not upload the data, please make sure to complete the previous recipe.

...

Loading data with Microsoft Fabric Dataflows

Fabric Data Factory has another data transformation tool in its arsenal: Dataflow Gen2 . Readers who have worked with Power BI will be familiar with the concept of Dataflows. While there are many similarities between Data Factory Dataflow Gen2 and Power BI Dataflows (they both allow users to ingest, transform, and combine data from various sources to create reusable and scalable data preparation processes), it’s important to note that as of the time of writing, they are not interchangeable.

If you went through previous chapters of this book, you will remember that ADF also has a concept of Dataflows – a tool to perform common data transformations without writing code. In Fabric, Dataflow Gen2 is considered a Data Factory component, but it has a distinctive interface and allows users to leverage Power Query when designing transformations.

We shall use an instance of Dataflow Gen2 to import data from a data...

Join our book community on Discord

https://packt.link/p3FAF

The Azure ecosystem is comprised of variety of different services. Most of them can be integrated and connected into Azure Data Factory (ADF). In this chapter, we will show you how to harness the power of serverless execution by integrating some of the most commonly used Azure services: Azure Logic Apps and Azure Functions. These recipes will help you understand how Azure services can be useful in designing Extract, Transform, Load (ETL) pipelines.We will cover the following recipes in this chapter:

• Triggering your data processing with Logic Apps
• Using the web activity to call an Azure logic app
• Adding flexibility to your pipelines with Azure Functions

Technical requirements

For this chapter, you will need the following:

• An active Azure account: It could be either your business account or a personal account. If you don't have an Azure account yet, you can activate an Azure free-trial license through Microsoft's main website: https://azure.microsoft.com/en-us/free/.
• GitHub repository: You can download the dataset from this book's GitHub repo or you can use your own: https://github.com/PacktPublishing/Azure-Data-Factory-Cookbook/tree/master/data.