Real-Time Data Analysis - Azure Stream Analytics

While some Azure components enable us to deliver data to the cloud, in most cases we also need something that is designed for analyzing and querying streamed data. One such service is Azure Stream Analytics, a real-time data analysis tool, which is able to read all messages sent through, for example, Event Hub, and transform, and save them using one of the predefined outputs.

The following topics will be covered in this chapter:

• Working with Azure Stream Analytics
• Available input and output types
• Querying data using the query language
• Ensuring the correct order of incoming data and performing checkpoints or replays

Technical requirements

To perform the exercises in this chapter, you will need:

• Visual Studio 2017 instance
• An Azure subscription
• Azure Stream Analytics tools—https://docs.microsoft.com/en-us/azure/stream-analytics/stream-analytics-tools-for-visual-studio-install

Azure Stream Analytics introduction

In the previous chapter, we discussed Azure Event Hub, which is a solution for receiving and processing thousands of messages per second, by introducing the implementation of event processor hosts. While it is great for workloads such as big data pipelines or IoT scenarios, it is not a solution to everything, especially if you want to avoid hosting VMs. Scaling such architectures can be cumbersome and nonintuitive; this is why there is Azure Stream Analytics, which is an event-processing engine designed for high volumes of data. It fills a gap where other services such as Event Hub or IoT Hub do not perform well (or where to do so they require much more skill and/or more sophisticated architecture), particularly for real-time analytics, anomaly detection, and geospatial analytics. It is an advanced tool for advanced tasks, which will greatly...

Input and output types

Azure Stream Analytics offers a seamless integration with some native Azure services, such as Azure Event Hub, Azure IoT Hub, or Azure Blob Storage. Additionally, it can be easily configured to output data to an SQL database, Blob, or Event Azure Data Lake Store. To leverage those possibilities, you will have to define both input and output types, which you are interested in. This allows for data to be easily ingested (in the form of a stream), so a job, which you will write, can work on thousands of events, analyzing and processing them. In this section, you will learn how to get started with Azure Stream Analytics and to define both the inputsand outputs.

Create Azure Stream Analytics in Azure portal

...

Azure Stream Analytics query language

The strength of Azure Stream Analytics, besides the rich selection of Azure services that seamlessly integrate with it, lies in its query language, which allows you to analyze an input stream easily and output it to a required service. As it is an SQL-like language, it should be intuitive and easy to learn for most developers using this service. Even if you are not familiar with SQL, the many examples available and its simple syntax should make it easy for you.

Writing a query

In the Azure portal, the query window for Azure Stream Analytics can be found either in the Overview or Query blade:

In the preceding example, you can see a simple SQL-like query, which performs the following three...

Event ordering, checkpoints, and replays

In the previous sections, we covered some basic topics of Azure Stream Analytics: how to configure inputs and outputs, querying data, and using the service. In the last part of this chapter, I will show you its more advanced features such as event ordering, checkpoints, and replays, which ensure that events are processed exactly in a way you would expect. These topics are in fact common subjects in many different messaging solutions, so you will be able to use knowledge from this chapter in your other projects.

Event ordering

There are two concepts of events when it comes to their ordering:

• Application (or event) time
• Arrival time

There is a clear distinction between them:

• Application...

Summary

In this chapter, we covered Azure Stream Analytics, a service for processing streams of data in near real time. You have learned what the available inputs and outputs are and how to configure them. What is more, you were able to write your first query, and check how the query language works for analyzing and processing incoming events. If you need a PaaS that can quickly read and transform events and push them to many different Azure services, Azure Stream Analytics is for you.

In the next chapter, we will go through Azure Service Bus, an enterprise-class messaging solution that is in fact the foundation of Azure Event Hub, which we discussed previously.

Questions

1. What is the payment model for Azure Stream Analytics?
2. What is the difference between a stream and the reference output?
3. What is the difference between application and arrival time?
4. Which query construct do you need to select an ID from an input and push it to an output?
5. Can you process different inputs in the same query?
6. When is an event considered out of order?
7. Is it possible to get a substring from a property in a query? If so, which function can be used for that?

Further reading

• Scaling and SUs: https://docs.microsoft.com/en-us/azure/stream-analytics/stream-analytics-streaming-unit-consumption
• Different output types: https://docs.microsoft.com/en-us/azure/stream-analytics/stream-analytics-define-outputs
• Common query patterns: https://docs.microsoft.com/en-us/azure/stream-analytics/stream-analytics-stream-analytics-query-patterns
• Window functions: https://docs.microsoft.com/en-us/azure/stream-analytics/stream-analytics-window-functions