Investigating IoT Data with Azure Data Explorer

In today’s data-driven world, organizations are faced with an overwhelming amount of information flowing through their systems. This is especially true of many IoT and logging solutions. To stay ahead of the game, it is crucial to efficiently collect, store, analyze, and visualize this data to extract valuable insights. This is where Azure Data Explorer (ADX) comes into play, which is a lightning-fast data analytics service provided by Microsoft Azure.

In this chapter, we will dive into the world of ADX and explore its capabilities, its features, and how it can revolutionize the way you work with data. We will walk you through the essential concepts and a hands-on example that will help you understand how ADX is used. While we will focus on how it is used for analyzing IoT data, it can also be used with a vast amount of different data types and sources.

Here’s an overview of what you can expect to learn in this chapter...

What is Azure Data Explorer?

Azure Data Explorer (ADX) is an advanced and fully managed analytics platform specifically designed for processing vast amounts of data with incredible speed and efficiency. With its comprehensive set of tools and features, ADX offers a complete solution for data ingestion, querying, visualization, and management.

By analyzing diverse data types such as structured, semi-structured, and unstructured data across time series, ADX empowers users to extract valuable insights, identify patterns and trends, and even develop accurate forecasting models. It incorporates the power of machine learning, making complex analytical tasks remarkably simple.

Scalability, security, and robustness are at the core of ADX, making it an ideal choice for a wide range of applications. Whether you need to perform log analytics, time series analytics, IoT data analysis, or general-purpose exploratory analytics, ADX is a reliable and enterprise-ready solution.

ADX is built...

Ingesting streaming data

Although you can ingest a variety of data into Data Explorer for analysis, for our purposes we are going to look at ingesting IoT data streams.

Streaming ingestion is valuable for data loading when there is a need for minimal delay between the process of ingesting data and querying it. It is advisable to employ streaming ingestion in the following situations:

• When a latency of less than one second is necessary
• To enhance the operational processing of numerous tables where the flow of data into each table is relatively small (a few records per second), while the overall data ingestion volume is substantial (thousands of records per second)
• If the flow of data into each table is substantial (over 4 GB per hour), it is recommended to use batch ingestion

There are two types of ingestion supported in ADX: data connection and custom ingestion:

• Data connection: You can use this if the data is coming from Event Hubs, IoT Hub, and Event...

Visualizing the streaming data

The visualization and reporting of data play a vital role in the data analytics workflow. ADX enables the development of advanced analytics solutions to handle extensive data volumes. By seamlessly integrating with multiple visualization tools, ADX allows you to effectively present and distribute data insights throughout your organization. These insights can be converted into actionable information to drive meaningful outcomes for your business.

Along with its own dashboarding capabilities, ADX integrates with other visualization tools such as Power BI, Excel, Grafana, Tableau, and Qlik, to name just a few. In addition, ADX provides native advanced analytics for time series analysis, pattern recognition, anomaly detection, and forecasting.

To create a dashboard in ADX, navigate to the ADX web UI and select the Dashboard tab in the left-side menu.

Figure 8.4 – Creating a dashboard

You get the option of creating...

Lab – creating an ADX dashboard

In this lab, we will explore how to consume and analyze IoT data using ADX. We will then leverage ADX’s capabilities to create a real-time dashboard that visualizes the IoT data:

1. Set up IoT Hub and a device capturing data from the Raspberry Pi online simulator (https://azure-samples.github.io/raspberry-pi-web-simulator/#getstarted).
2. Verify that the signal is being sent to IoT Hub.
3. Set up an SDX cluster and make sure you enable streaming data.
4. Create a database and a table in ADX to capture the data.
5. Define the table mapping between the incoming messages and the table columns.
6. Create a new dashboard and enter the following KQL using your table name:

   Telemetry
    | where IotHubEnqueuedTime between (['_startTime'] .. ['_endTime']) // Time range filtering

7. After a few minutes, you should see results when you run the KQL.
8. Select Add visual next to the Results tab.
9. Add a line chart showing...

Summary

In this chapter, you were introduced to ADX, which is an excellent tool to use when exploring and analyzing the data collected from IoT devices. With the proliferation of IoT devices in various industries, there is an immense amount of data being generated, and the ability to extract valuable insights from this data is crucial.

The chapter began by introducing ADX and explaining its role in handling large volumes of streaming data. It highlighted ADX’s capabilities in ingesting, storing, and querying data in real time, making it suitable for scenarios involving IoT devices, log analytics, and other streaming data sources. Next, the chapter dug into the process of ingesting streaming data into ADX. It provided guidance on configuring data ingestion pipelines.

Another crucial aspect of streaming data analysis is data visualization, and the chapter explored how ADX enables users to create compelling visual representations of their data. To provide you with hands-on...

Further reading

This book is worth looking at if you wish to delve deeper into ADX and KQL:

• Scalable Data Analytics with Azure Data Explorer, Packt Publishing, by Jason Myerscough