Big Data Pipeline - Azure Event Hub

Azure Event Hub is one of the best solutions for introducing an entry point with almost limitless throughput. It's designed for big data workloads and is able to process millions of messages per second. It offers a very simple configuration, and thanks to the available SDK, you can easily adjust it to almost any solution developed in the cloud. It also integrates natively with other Azure components, making creating a whole platform hosted in the cloud a breeze.

The following topics will be covered in this chapter:

• Working efficiently with Azure Event Hub
• Different concepts such as publishers, partitions, throughput units, or consumer groups
• Azure Event Hub security concepts
• Azure Event Hub capture feature

Technical requirements

To perform the exercises in this chapter, you will need the following:

• Microsoft Azure subscription
• Visual Studio 2017

Azure Event Hub service and concepts

Nowadays, we gather more and more data, which has to be aggregated, processed, and stored somewhere. This implies using services that can handle increasing loads, scale to growing demands, and offer the smallest latency available. All these requirements are often mentioned when building so-called big data pipelines—parts of a system designed to process as much data as possible, so it is later accessible by tools such as Hadoop, Spark, ML, AI, and so on. If you are looking for a service in Azure that can handle millions of messages per second, Azure Event Hub is the right choice. In this chapter, you will learn the basics of this Azure component and get familiar with messaging solutions in Azure.

Azure Event Hub concepts

...

Working with Azure Event Hub

Now that you are familiar with some basic concepts, we can proceed and start working with a real instance of Azure Event Hub. In this section, you will learn both how to create and access Event Hub in the Azure portal and work with it using its SDK. In fact, using this service is possible both from the portal (as many Azure components seamlessly integrate with it and no additional configuration is required) and by providing custom implementations of consumers that read and process data further.

Creating an Azure Event Hub in the Azure portal

To create an Azure Event Hub, we will start, as in most cases, by clicking on the + Create a resource button. Enter Event Hub and select the service from...

Azure Event Hub security

We have covered some topics regarding working with and developing applications using Azure Event Hub – now it is time to learn something more about the security features of this service. In the previous part of this chapter, you used shared access policies, which are the easiest options when you want to restrict access to a hub to some predefined operations (such as listening to events, sending them, or managing Event Hub). Now I will show you something more about the security model and restricting access to the whole namespace by IP filtering.

Event publishers

When creating an Event Hub namespace instance, you had to choose a tier – you could choose between Basic and Standard tier. Besides...

Azure Event Hub Capture feature

There is one feature of Azure Event Hub that requires an individual section itself to describe it in depth. It is capture, a functionality which allows you to automatically store events using a predefined storage solution (such as Azure Storage or Azure Data Lake) and process it further. Unfortunately, this particular feature is often misused as its use cases are not so obvious; additionally, the way it works might sometimes be unclear.

How Azure Event Hub Capture works

In common use cases for Event Hub, you need a producer and a consumer to fetch data and process it. Let's consider the following scenario:

In this scenario, we have two consumers:

• Consumer 1 for some generic processing...

Summary

In this chapter, you have learned many things about Azure Event Hub—how it works, what partitions are for, and how to leverage more advanced features such as consumer groups or the Event Hub Capture feature. I strongly encourage you to give it a try and play a little bit with this Azure service as it is a powerful tool for processing thousands of events per second. It is also pretty simple to use and does not require much time to get started.

In the next chapter, you will learn about another service for processing many events, and additionally analyze and transform them in near real time—Azure Stream Analytics.

Questions

1. What are consumer groups for?
2. How many events can be processed in one second using 1 TU?
3. How many partitions should you use for each Event Hub?
4. Are TUs assigned to a namespace or a particular Event Hub?
5. What are the three different permissions you can assign to an access policy?
6. Can an event publisher listen to incoming events using its token?
7. What happens if you have more consumers than partitions?

Further reading

The full documentation on Event Hub disaster recovery of can be found here: https://docs.microsoft.com/en-us/azure/event-hubs/event-hubs-geo-dr.

Apache Avro Documentation can be found here: https://avro.apache.org/.