Home Data Extending Power BI with Python and R - Second Edition

Extending Power BI with Python and R - Second Edition

By Luca Zavarella
ai-assist-svg-icon Book + AI Assistant
eBook + AI Assistant $43.99
Print $54.99
Subscription $15.99
ai-assist-svg-icon NEW: AI Assistant (beta) Available with eBook, Print, and Subscription.
ai-assist-svg-icon NEW: AI Assistant (beta) Available with eBook, Print, and Subscription. $10 p/m for first 3 months. $15.99 p/m after that. Cancel Anytime! ai-assist-svg-icon NEW: AI Assistant (beta) Available with eBook, Print, and Subscription.
What do you get with a Packt Subscription?
Gain access to our AI Assistant (beta) for an exclusive selection of 500 books, available during your subscription period. Enjoy a personalized, interactive, and narrative experience to engage with the book content on a deeper level.
This book & 7000+ ebooks & video courses on 1000+ technologies
60+ curated reading lists for various learning paths
50+ new titles added every month on new and emerging tech
Early Access to eBooks as they are being written
Personalised content suggestions
Customised display settings for better reading experience
50+ new titles added every month on new and emerging tech
Playlists, Notes and Bookmarks to easily manage your learning
Mobile App with offline access
What do you get with a Packt Subscription?
This book & 6500+ ebooks & video courses on 1000+ technologies
60+ curated reading lists for various learning paths
50+ new titles added every month on new and emerging tech
Early Access to eBooks as they are being written
Personalised content suggestions
Customised display settings for better reading experience
50+ new titles added every month on new and emerging tech
Playlists, Notes and Bookmarks to easily manage your learning
Mobile App with offline access
What do you get with eBook + Subscription?
Download this book in EPUB and PDF formats, plus a monthly download credit
This book & 6500+ ebooks & video courses on 1000+ technologies
60+ curated reading lists for various learning paths
50+ new titles added every month on new and emerging tech
Early Access to eBooks as they are being written
Personalised content suggestions
Customised display settings for better reading experience
50+ new titles added every month on new and emerging tech
Playlists, Notes and Bookmarks to easily manage your learning
Mobile App with offline access
What do you get with a Packt Subscription?
Gain access to our AI Assistant (beta) for an exclusive selection of 500 books, available during your subscription period. Enjoy a personalized, interactive, and narrative experience to engage with the book content on a deeper level.
This book & 6500+ ebooks & video courses on 1000+ technologies
60+ curated reading lists for various learning paths
50+ new titles added every month on new and emerging tech
Early Access to eBooks as they are being written
Personalised content suggestions
Customised display settings for better reading experience
50+ new titles added every month on new and emerging tech
Playlists, Notes and Bookmarks to easily manage your learning
Mobile App with offline access
What do you get with eBook?
Along with your eBook purchase, enjoy AI Assistant (beta) access in our online reader for a personalized, interactive reading experience.
Download this book in EPUB and PDF formats
Access this title in our online reader
DRM FREE - Read whenever, wherever and however you want
Online reader with customised display settings for better reading experience
What do you get with video?
Download this video in MP4 format
Access this title in our online reader
DRM FREE - Watch whenever, wherever and however you want
Online reader with customised display settings for better learning experience
What do you get with video?
Stream this video
Access this title in our online reader
DRM FREE - Watch whenever, wherever and however you want
Online reader with customised display settings for better learning experience
What do you get with Audiobook?
Download a zip folder consisting of audio files (in MP3 Format) along with supplementary PDF
What do you get with Exam Trainer?
Flashcards, Mock exams, Exam Tips, Practice Questions
Access these resources with our interactive certification platform
Mobile compatible-Practice whenever, wherever, however you want
ai-assist-svg-icon NEW: AI Assistant (beta) Available with eBook, Print, and Subscription. ai-assist-svg-icon NEW: AI Assistant (beta) Available with eBook, Print, and Subscription. BUY NOW $10 p/m for first 3 months. $15.99 p/m after that. Cancel Anytime! ai-assist-svg-icon NEW: AI Assistant (beta) Available with eBook, Print, and Subscription.
eBook + AI Assistant $43.99
Print $54.99
Subscription $15.99
What do you get with a Packt Subscription?
Gain access to our AI Assistant (beta) for an exclusive selection of 500 books, available during your subscription period. Enjoy a personalized, interactive, and narrative experience to engage with the book content on a deeper level.
This book & 7000+ ebooks & video courses on 1000+ technologies
60+ curated reading lists for various learning paths
50+ new titles added every month on new and emerging tech
Early Access to eBooks as they are being written
Personalised content suggestions
Customised display settings for better reading experience
50+ new titles added every month on new and emerging tech
Playlists, Notes and Bookmarks to easily manage your learning
Mobile App with offline access
What do you get with a Packt Subscription?
This book & 6500+ ebooks & video courses on 1000+ technologies
60+ curated reading lists for various learning paths
50+ new titles added every month on new and emerging tech
Early Access to eBooks as they are being written
Personalised content suggestions
Customised display settings for better reading experience
50+ new titles added every month on new and emerging tech
Playlists, Notes and Bookmarks to easily manage your learning
Mobile App with offline access
What do you get with eBook + Subscription?
Download this book in EPUB and PDF formats, plus a monthly download credit
This book & 6500+ ebooks & video courses on 1000+ technologies
60+ curated reading lists for various learning paths
50+ new titles added every month on new and emerging tech
Early Access to eBooks as they are being written
Personalised content suggestions
Customised display settings for better reading experience
50+ new titles added every month on new and emerging tech
Playlists, Notes and Bookmarks to easily manage your learning
Mobile App with offline access
What do you get with a Packt Subscription?
Gain access to our AI Assistant (beta) for an exclusive selection of 500 books, available during your subscription period. Enjoy a personalized, interactive, and narrative experience to engage with the book content on a deeper level.
This book & 6500+ ebooks & video courses on 1000+ technologies
60+ curated reading lists for various learning paths
50+ new titles added every month on new and emerging tech
Early Access to eBooks as they are being written
Personalised content suggestions
Customised display settings for better reading experience
50+ new titles added every month on new and emerging tech
Playlists, Notes and Bookmarks to easily manage your learning
Mobile App with offline access
What do you get with eBook?
Along with your eBook purchase, enjoy AI Assistant (beta) access in our online reader for a personalized, interactive reading experience.
Download this book in EPUB and PDF formats
Access this title in our online reader
DRM FREE - Read whenever, wherever and however you want
Online reader with customised display settings for better reading experience
What do you get with video?
Download this video in MP4 format
Access this title in our online reader
DRM FREE - Watch whenever, wherever and however you want
Online reader with customised display settings for better learning experience
What do you get with video?
Stream this video
Access this title in our online reader
DRM FREE - Watch whenever, wherever and however you want
Online reader with customised display settings for better learning experience
What do you get with Audiobook?
Download a zip folder consisting of audio files (in MP3 Format) along with supplementary PDF
What do you get with Exam Trainer?
Flashcards, Mock exams, Exam Tips, Practice Questions
Access these resources with our interactive certification platform
Mobile compatible-Practice whenever, wherever, however you want
  1. Free Chapter
    Where and How to Use R and Python Scripts in Power BI
About this book
The latest edition of this book delves deep into advanced analytics, focusing on enhancing Python and R proficiency within Power BI. New chapters cover optimizing Python and R settings, utilizing Intel's Math Kernel Library (MKL) for performance boosts, and addressing integration challenges. Techniques for managing large datasets beyond available RAM, employing the Parquet data format, and advanced fuzzy matching algorithms are explored. Additionally, it discusses leveraging SQL Server Language Extensions to overcome traditional Python and R limitations in Power BI. It also helps in crafting sophisticated visualizations using the Grammar of Graphics in both R and Python. This Power BI book will help you master data validation with regular expressions, import data from diverse sources, and apply advanced algorithms for transformation. You'll learn how to safeguard personal data in Power BI with techniques like pseudonymization, anonymization, and data masking. You'll also get to grips with the key statistical features of datasets by plotting multiple visual graphs in the process of building a machine learning model. The book will guide you on utilizing external APIs for enrichment, enhancing I/O performance, and leveraging Python and R for analysis. You'll reinforce your learning with questions at the end of each chapter.
Publication date:
March 2024
Publisher
Packt
Pages
814
ISBN
9781837639533

 

Where and How to Use R and Python Scripts in Power BI

Power BI is Microsoft’s flagship self-service business intelligence product. It consists of a set of on-premises applications and cloud-based services that help organizations integrate, transform, and analyze data from a wide variety of source systems through a user-friendly interface.

The platform is not limited to data visualization. Power BI is much more than this when you consider that its analytics engine (Vertipaq) is the same as SQL Server Analysis Services (SSAS), Azure Analysis Services, and Power Pivot in Excel and it is also the engine used for reports and datasets published to the Power BI service. In addition, it uses Power Query as its data extraction and transformation engine, which we find in both Analysis Services and Excel. The engine comes with a very powerful and versatile formula language (M) and GUI, thanks to which you can “grind” and shape any type of data into any form.

Moreover, Power BI supports DAX as a data analytics formula language, which can be used for advanced calculations and queries on data that has already been loaded into tabular data models.

Such a versatile and powerful tool is a godsend for anyone who needs to do data ingestion and transformation in order to build dashboards and reports to summarize a company’s business.

Recently, the availability of huge amounts of data, along with the ability to scale the computational power of machines, has made the area of advanced analytics more appealing. So, new mathematical and statistical tools have become necessary in order to provide rich insights. Hence the integration of analytical languages such as Python and R within Power BI.

R or Python scripts can only be used within Power BI with specific features. Knowing which Power BI tools can be used to inject R or Python scripts into Power BI is key to understanding whether the problem you want to address is achievable with these analytical languages.

This chapter will cover the following topics:

  • Injecting R or Python scripts into Power BI
  • Using R and Python to interact with your data
  • Python and R compatibility across Power BI products
 

Technical requirements

This chapter requires you to have Power BI Desktop already installed on your machine (you can download it here: https://aka.ms/pbiSingleInstaller). The version used in this chapter is 2.110.1161.0 64-bit (October 2022).

 

Injecting R or Python scripts into Power BI

In this first section, Power BI Desktop tools that allow you to use Python or R scripts will be presented and described in detail. Specifically, you will see how to add your own code during the data loading, data transforming, and data viewing phases.

Data loading

One of the first steps required to work with data in Power BI Desktop is to import it from external sources:

There are many connectors that allow you to do this, depending on the respective data sources, but you can also do it via scripts in Python and R. In fact, if you click on the Get data icon in the ribbon, not only are the most commonly used connectors shown but you can also select other ones from a more complete list by clicking on More...:

Graphical user interface, text, application  Description automatically generated

Figure 1.1: Browse more connectors to load your data

In the new Get Data window that pops up, simply start typing the word script into the search box, and immediately the two options for importing data via Python or R appear:

Figure 1.2 – Showing R script and Python script into the Get Data window

Figure 1.2: Showing R script and Python script in the Get Data window

Reading the contents of the tooltip, obtained by hovering the mouse over the Python script option, two things should immediately jump out at you:

  1. A local installation of Python is required.
  2. What can be imported through Python is a data frame.

The same two observations also apply when selecting R script. The only difference is that it is possible to import a pandas DataFrame when using Python (a DataFrame is a data structure provided by the pandas package), whereas R employs the two-dimensional array-like data structure called an R data frame, which is provided by default by the language.

After clicking on the Python script option, a new window will be shown containing a text box for writing the Python code:

Figure 1.3 – Window showing the Python script editor

Figure 1.3: Window showing the Python script editor

As you can see, it’s definitely a very skimpy editor, but in Chapter 3, Configuring Python with Power BI, you’ll discover how you can utilize your preferred IDE to create your scripts within a more comprehensive and feature-rich editor.

Taking a look at the warning message, Power BI reminds you that no Python engine has been detected, so it must be installed. If you already have Python installed and configured, you will not see this message. Clicking on the How to install Python link will cause a Microsoft Docs web page to open, explaining the steps to install Python.

Microsoft suggests installing the base Python distribution, but in order to follow some best practices on environments (self-contained spaces that allow developers to manage dependencies, libraries, and configurations specific to individual projects), we will install the Miniconda distribution. The details of how to do this and why will be covered in Chapter 3.

If you had clicked on R script instead, a window for entering code in R, similar to the one shown in Figure 1.4, would have appeared:

Figure 1.4 – Window showing the R script editor

Figure 1.4: Window showing the R script editor

As with Python, in order to run code in R, you need to install the R engine on your machine. Clicking on the How to install R link will open a Docs page where Microsoft suggests installing either Microsoft R Open or the classic CRAN R. Chapter 2, Configuring R with Power BI, will show you which engine to choose and how to configure your favorite IDE to write code in R.

In order to import data using Python or R, you need to write code in the editors shown in Figure 1.3 and Figure 1.4, which assign a pandas DataFrame or an R data frame to a variable, respectively. You will see concrete examples throughout this book.

Next, let’s look at transforming data.

Data transformation

It is possible to apply a transformation to data already imported or being imported, using scripts in R or Python. Should you want to test this on the fly, you can import the following CSV file directly from the web: http://bit.ly/iriscsv. Follow these steps:

  1. Simply click on Get data and then Web to import data directly from a web page:
Graphical user interface, application  Description automatically generated

Figure 1.5: Select the Web connector to import data from a web page

  1. You can now enter the previously mentioned URL in the window that pops up:
Figure 1.6 – Import the Iris data from the web

Figure 1.6: Import the Iris data from the web

  1. Right after clicking OK, a window will pop up with a preview of the data to be imported.

    In this case, instead of importing the data as is, click on Transform Data in order to access the Power Query data transformation window:

    Table  Description automatically generated

    Figure 1.7: Imported data preview

    It is at this point that you can add a transformation step using a Python or R script by selecting the Transform tab in Power Query Editor:

    Graphical user interface, table, Excel  Description automatically generated

    Figure 1.8: R and Python script tools in Power Query Editor

    By clicking on Run Python script, you’ll cause a window, similar to the one you’ve already seen in the data import phase, to pop up:

    Graphical user interface, text, application, email  Description automatically generated

    Figure 1.9: The Run Python script editor

    If you carefully read the comment in the text box, you will see that the dataset variable is already initialized and contains the data present at that moment in Power Query Editor, including any transformations already applied. At this point, you can insert your Python code in the text box to transform the data into the desired form.

    A similar window will open if you click on Run R script:

    Graphical user interface, text, application, email  Description automatically generated

    Figure 1.10: The Run R script editor

Also, in this case, the dataset variable is already initialized and contains the data present at that moment in Power Query Editor. You can then add your own R code and reference the dataset variable to transform your data in the most appropriate way.

Next, let’s look at visualizing data.

Data visualization

Finally, your own Python or R scripts can be added to Power BI to create new visualizations, in addition to those already present in the tool out of the box:

  1. Assuming we resume the data import activity started in the previous section, once the Iris dataset is loaded, simply click Cancel in the Run R script window, and then click Close & Apply in the Home tab of Power Query Editor:
Figure 1.11 – Click Close & Apply to import the Iris data

Figure 1.11: Click Close & Apply to import the Iris data

  1. After the data import is complete, you can select either the R visual or Python visual option in the Visualizations pane of Power BI:
    Diagram  Description automatically generated

    Figure 1.12: The R and Python script visuals

    If you click on Python visual, a window pops up asking for permission to enable script code execution, as there may be security or privacy risks:

    Figure 1.13 – Enable the script code execution

    Figure 1.13: Enable the script code execution

  1. After enabling code execution, in Power BI Desktop, you can see a placeholder for the Python visual image on the report canvas and a Python script editor pane at the bottom:
Graphical user interface, application  Description automatically generated

Figure 1.14: The Python visual layout

Once you drag the fields you want to use in your Python script into the Values area, you can write your own custom code into the Python script editor and run it to generate a Python visualization.

A pretty much identical layout occurs when you select R visual.

 

Using R and Python to interact with your data

In the previous section, you saw all the ways you can interact with your data in Power BI via R or Python scripts. Beyond knowing how and where to inject your code into Power BI, it is very important to know how your code will interact with that data. It’s here that we see a big difference between the effect of scripts injected via Power Query Editor and scripts used in visuals:

  • Scripts via Power Query Editor: This type of script will transform the data and persist transformations in the model. This means that it will always be possible to retrieve the transformed data from any object within Power BI. Also, once the scripts have been executed and have taken effect, they will not be re-executed unless the data is refreshed. Therefore, it is recommended to inject code in R or Python via Power Query Editor when you intend to use the resulting insights in other visuals, or in the data model.
  • Scripts in visuals: The scripts used within the R and Python script visuals extract particular insights from the data and only make them evident to the user through visualization. Like all the other visuals on a report page, the R and Python script visuals are also interconnected with the other visuals. This means that the script visuals are subject to cross-filtering and therefore, they are refreshed every time you interact with other visuals in the report. That said, it is not possible to persist the results obtained from the script visuals in the data model.

TIP

Thanks to the interactive nature of R and Python script visuals due to cross-filtering, it is possible to inject code that is useful for extracting real-time insights from data. The important thing to keep in mind is that, as previously stated, it is then only possible to visualize such information, or at the most, to write it to external repositories (as you will see in Chapter 8, Logging Data from Power BI to External Sources). Also, although it is possible to access resources on the internet from a visual script when developing in Power BI Desktop, it is no longer possible to do so when the report is published to the Power BIs Service (you will see what this is about in the next section) due to security issues. This restriction doesn’t exist for scripts used in Power Query.

In the final section of this chapter, let’s look at the limitations of using R and Python when it comes to various Power BI products.

 

Python and R compatibility across Power BI products

The first question once you are clear on where to inject R and Python scripts in Power BI could be: “Is the use of R and Python code allowed in all Power BI products?” In order to cover that, let’s briefly recap the various Power BI products and their usage in general. Here is a concise list:

  • Power BI service: This is sometimes called Power BI Online, and it’s the Software as a Service (SaaS) version of Power BI. It was created to facilitate the sharing of visual analysis between users through dashboards and reports.
  • Power BI Report Server: This is the on-premises version of Power BI and it extends the capabilities of SQL Server Reporting Services, enabling the sharing of reports created in Power BI Desktop (for Report Server) and Power BI Report Builder for Power BI paginated reports.
  • Power BI embedded: A Microsoft Azure service that allows dashboards and reports to be embedded in an application for users who do not have a Power BI account.
  • Power BI Desktop: A free desktop application for Windows that allows you to use almost all of the features that Power BI offers. It is not the right tool for sharing results between users, but it allows you to share them on the Power BI service and Power BI Report Server. The desktop versions that allow publishing on the two mentioned services are distinct and support slightly different sets of features. They are named Power BI Desktop and Power BI Desktop for Power BI Report Server, respectively.
  • Power BI mobile: A mobile application, available on Windows, Android, and iOS, that allows secure access to the Power BI service and Power BI Report Server, and that allows you to browse and share dashboards and reports, but not edit them.
  • Power BI Report Builder: A free desktop application for Windows that allows you to create paginated reports. These can then be published and shared in the Power BI service and Power BI Report Server.

Apart from the licenses, which we will not go into here, a summary figure of the relationships between the previously mentioned products follows:

Diagram  Description automatically generated

Figure 1.15: Interactions between Power BI products

Unfortunately, of all these products, only the Power BI service, Power BI Embedded, and Power BI Desktop allow you to enrich data via code in R and Python:

Diagram  Description automatically generated with low confidence

Figure 1.16: Power BI products, compatibility with R and Python

IMPORTANT NOTE

From here on out, when we talk about the Power BI service in terms of compatibility with analytical languages, what we say will also apply to Power BI embedded.

So, if you need to develop reports using advanced analytics through R and Python, make sure the target platform supports them.

 

Summary

This chapter has given a detailed overview of all the ways in which you can use R and Python scripts in Power BI Desktop. During the data ingestion and data transformation phases, Power Query Editor allows you to add steps containing R or Python code. You can also make use of these analytical languages during the data visualization phase thanks to the R and Python script visuals provided by Power BI Desktop.

It is also very important to know how the R and Python code will interact with the data already loaded or being loaded in Power BI. If you use Power Query Editor, both when loading and transforming data, the result of script processing will be persisted in the data model. Also, if you want to run the same scripts again, you have to refresh the data. On the other hand, if you use the R and Python script visuals, the code results can only be displayed and are not persisted in the data model. In this case, script execution occurs whenever cross-filtering is triggered via the other visuals in the report.

Unfortunately, at the time of writing, you cannot run R and Python scripts in every Power BI product. The only ones that provide for running analytics scripts are Power BI Desktop and the Power BI service.

In the next chapter, we will see how best to configure the R engine and RStudio to integrate with Power BI Desktop.

 

Test your knowledge

  1. At what stages of Power BI report development can scripts in Python or R be used?
  2. Is it possible to use a Python dictionary as a data source for a Power BI report?
  3. Is it possible to use an R list as a data source for a Power BI report?
  4. When you insert a Python or R script step immediately after other transformation steps that return a specific result set, what is the name of the variable you will need to use in your script to access the data obtained in the step just before?
  5. After adding a Python or R visual script to your canvas, what do you need to do to enable the script editor to take a script that generates a plot?
  6. What are the ways by which to force the re-execution of Python or R scripts added via Power Query?
  7. What are the ways by which to force the re-execution of Python or R scripts added in a script visual?
  8. In what case is it not possible to access the internet from a Python or R script in Power BI?
  9. In which Power BI products can Python or R scripts be used?

Learn more on Discord

To join the Discord community for this book – where you can share feedback, ask questions to the author, and learn about new releases – follow the QR code below:

https://discord.gg/MKww5g45EB

About the Author
  • Luca Zavarella

    Luca Zavarella has a rich background as an Azure Data Scientist Associate and Microsoft MVP, with a Computer Engineering degree from the University of L'Aquila. His decade-plus experience spans the Microsoft Data Platform, starting as a T-SQL developer on SQL Server 2000 and 2005, then mastering the full suite of Microsoft Business Intelligence tools (SSIS, SSAS, SSRS), and advancing into data warehousing. Recently, his focus has shifted to advanced analytics, data science, and AI, contributing to the community as a speaker and blogger, especially on Medium. Currently, he leads the Data & AI division at iCubed, and he also holds an honors degree in classical piano from the "Alfredo Casella" Conservatory in L'Aquila.

    Browse publications by this author
Extending Power BI with Python and R - Second Edition
Unlock this book and the full library FREE for 7 days
Start now