Exploring Azure Stream Analytics

Azure Stream Analytics (ASA) is a stream processing engine that is intended to analyze and process large volumes of streaming data with sub-millisecond latencies. It includes a robust SQL-like query language that you can use to query your streaming data to detect anomalies, aggregate data over different time windows, and perform geospatial analysis for moving sensors.

In this chapter, you will learn how to incorporate this vital tool as part of your stream processing. To fully comprehend how Stream Analytics processes data, we will dissect its inputs and outputs. We’ll uncover how data is ingested from an IoT device and transformed into actionable insights, allowing organizations to react swiftly to changing conditions.

As we venture further into the world of Stream Analytics, we’ll acquaint ourselves with the Stream Analytics query language. This language serves as the backbone of the system, providing a robust toolkit for filtering...

Stream analytics use cases

Stream analytics, also known as real-time analytics or real-time data processing, involves analyzing and deriving insights from data that is generated continuously and in real time. Here are some common use cases for stream analytics:

• Fraud detection: Stream analytics can be used to detect fraudulent activities in real time. By continuously analyzing incoming data from various sources, such as financial transactions or user behavior patterns, patterns indicative of fraud can be identified and appropriate actions can be taken immediately.
• IoT data processing: IoT generates vast amounts of data from connected devices. Stream analytics can process and analyze this data in real time, enabling real-time monitoring, anomaly detection, predictive maintenance, and operational optimization.
• Network monitoring and anomaly detection: Stream analytics can be used to monitor network traffic and detect anomalies or suspicious activities in real time. This...

Inputs and outputs

Before we can look at inputs and outputs, we need to create an ASA job. In your resource group, select Add a resource and search for azure stream analytics:

Figure 7.1 – Creating a cloud ASA job

While creating a Stream Analytics job service, you will see you can host it in the cloud or on the edge. For now, we are going to host it in the cloud. You also need to set the number of streaming units (SUs). When you create a Stream Analytics job in Azure, you can configure the number of SUs to allocate to that job. The number of Azure Scale Units (ASUs) determines the job’s processing capacity, and it affects how much data the job can process in real time. For demo purposes, we can just use the default of three units:

Figure 7.2 – Setting up a Stream Analytics job

When setting up a Stream Analytics job, you first need to define an input. The two types of inputs are stream inputs and reference inputs...

Stream Analytics query language

ASQL simplifies the process of defining and executing real-time data processing logic on streaming data. Its SQL-like syntax, along with stream-specific constructs, enables developers and data engineers to perform complex analytics, transformations, and aggregations on streaming data with ease.

Here are some key features and concepts of ASQL:

• Input and output sources: ASQL allows you to define input sources, such as event hubs or IoT hubs, from which streaming data is ingested. You can specify the schema of the incoming data and define output sinks, such as Azure Blob Storage or Azure SQL Database, to store or forward the processed data.
• Query structure: ASQL queries typically follow a structure similar to SQL SELECT statements. However, ASQL queries operate on an infinite, unbounded stream of data, rather than discrete tables. The data is processed in an event-driven and continuous manner.
• Windowing: ASQL supports various windowing...

Running and monitoring jobs

After creating and testing a job, the next step is running and monitoring it to ensure it is processing data as expected and promptly detect any issues or anomalies.

Before running a job, you should set up the diagnostic settings by going to Diagnostic settings on the left side menu. The following screenshot demonstrates setting up execution logs saved to a storage account:

Figure 7.8 – Setting up diagnostic logs

In this case, we are looking at execution logs and retaining them for 10 days.

ASA also allows you to set up alerts based on specific conditions or metrics. By creating alerts, you can receive notifications when certain thresholds or conditions are met. In the Azure portal, go to your Streaming Analytics service, click on Alerts on the left-hand menu, and configure the alert rules based on your requirements.

Once the job is created, on the job overview page, click on the Start job button to start the job...

Lab – detecting anomalies with ASA

One of the most common use cases of ASA is to detect anomalies. In this lab, you will see how it is done:

1. Create an IoT hub with a device named device1.
2. Go to the device simulator at https://azure-samples.github.io/raspberry-pi-web-simulator and enter your device connection string.
3. Start the simulator and make sure your IoT hub is getting messages. You can use the following CLI command to view messages:

   az iot hub monitor-events --hub-name

4. If it asks you to install dependencies, choose Yes.
5. You can stop the simulator while you set up the ASA job.
6. Create a blob storage account to save the results of the job. Create a blob container named logs.
7. In the same resource group, select Create a resource and search for stream analytics job.

   Create a Stream Analytics job using the default values.

8. Once created, open the Stream Analytics job in the Azure portal and select Inputs on the left-hand menu. Add an IoT hub input...

Summary

In this chapter, the focus was on ASA, a powerful stream processing engine designed to handle and analyze vast amounts of streaming data with minimal latency. The chapter began by introducing the fundamental concepts and features of ASA.

The chapter emphasized the significance of using the robust SQL-like query language offered by ASA to efficiently query streaming data. This query language enables users to detect anomalies in the data, aggregate data over different time windows, and perform geospatial analysis for moving sensors. By mastering this query language, you can leverage the full potential of ASA for your stream processing needs.

Additionally, the chapter covered essential aspects such as setting up inputs and outputs for ASA. Understanding how to configure and manage these data sources and destinations is crucial for effective stream processing.

The next chapter delves into the capabilities of Azure Data Explorer, a fully managed, high-performance analytics...