Chapter 6: Data Architecture

In this chapter, we explore how data is processed and stored. We will look at specialized data stores that are uniquely tailored to each dataset and purpose. We will explore traditional Relational Database Management System (RDBMS) workloads, as well as modern big data solutions.

We will more specifically cover the following topics:

• Looking at the data architecture map
• Analyzing traditional data practices
• Delving into non-traditional data services
• Diving into big data and AI services
• Getting our hands dirty with a near real-time data streaming use case

This chapter will provide you with a detailed understanding of different data services and a good overview of big data and Artificial Intelligence (AI). You will also gain some hands-on experience of how to process data in real time with Azure.

Let's now review the technical requirements.

Technical requirements

If you want to practice the explanations provided in this chapter, you will need the following:

• An Azure subscription: To create your free Azure account, follow the steps explained at https://azure.microsoft.com/en-us/free/.
• A Power BI workspace (with a Power BI Pro license): To create a free Power BI account, follow the steps explained at https://app.powerbi.com/signupredirect?pbi_source=web.
• A shell system: This could be DOS, PowerShell, or even Bash on Linux. This is required to run the executable .NET Core app that we have built for you toward the end of this chapter. We will provide explanations on how to start the application with DOS.

All the code and maps (in full size) used in this chapter are available at https://github.com/PacktPublishing/The-Azure-Cloud-Native-Architecture-Mapbook/tree/master/Chapter06.

Let's start with our data architecture map.

The CiA videos for this book can be viewed at: http://bit.ly/3pp9vIH...

Looking at the data architecture map

It is no secret that every system and application deals with data, and it is no secret that the importance of data is growing year after year, especially with the rise of AI. Data volumes are higher than ever before. Companies need to find ways to store data efficiently and at a reasonable cost while being able to get insights from it. In this chapter, we will browse the vast data landscape of Azure. A reduced version of our data architecture map is shown in Figure 6.1:

Figure 6.1 – The data architecture map (reduced)

Important note

To see the full data architecture map (Figure 6.1), you can download the PDF file at https://github.com/PacktPublishing/The-Azure-Cloud-Native-Architecture-Mapbook/blob/master/Chapter06/maps/Data%20Architecture.pdf.

Our map has five top-level groups:

• BIG DATA: The public cloud is probably the only viable option that deals with real big data. However, big data services...

Analyzing traditional data practices

In this section, we will review services that belong to the traditional data world. Our purpose is to reassure you that what we can do on-premises can also be done the same way in the cloud. Moving data workloads to the cloud does not necessarily mean that you have to completely reinvent yourself and your way of working.

Let's first clarify that we do not use the term traditional in a pejorative (nor negative) way. In many situations and for many enterprises, using traditional techniques still provides full satisfaction, and you should not necessarily move to more modern technologies just for the sake of it. (Our point has been made!) Figure 6.2 displays our zoom-in on traditional data practices:

Figure 6.2 – Traditional data practices in Azure

The TRADITIONAL node regroups all traditional practices that we've typically used for decades, for which Azure also has a bunch of services that we will discuss...

Delving into modern data services and practices

As stated in the previous section, you should not rush to newer technologies for the sake of it. However, the good news is that most of these technologies are available at an affordable price, so you should at least explore them. Although we try to set services under a certain category, some of them span multiple categories. So, you shouldn't consider their positions on the map to be official or the only way to organize them. For example, Azure Storage spans at least the modern and big data categories. Figure 6.6 shows Azure's vast modern data landscape, which we will review in the upcoming subsections:

Figure 6.6 – The modern data landscape in Azure

Let's now explore the ELT category, which is the modern counterpart of ETL.

Introducing the ELT practice

ELT is quite similar to ETL, with one notable exception: the transformation step. As we saw earlier, in ETL, every step is entirely...

Diving into big data services

Big data goes way beyond traditional data technologies, so it could have been placed under the MODERN top-level group. However, as stated before, it is easier to make it a separate group for the sake of clarity. In a nutshell, big data deals with data sets that are too large to be handled by traditional data technologies. The good news, however, is that the opposite is possible. You can use big data technologies to handle smaller volumes, and it is doable at an affordable price in the cloud. Figure 6.10 shows the big data landscape in Azure:

Figure 6.10 – Big data in Azure

As a preamble, we can say that Azure's data service masterpiece is Azure Synapse Analytics (ASA). ASA can be used for whatever data purpose you may have with Azure. Therefore, it's impossible to position it in a very specific place on the map. So, if you know nothing about Azure data services, you should consider ASA an all-in-one data service...

Introducing AI solutions

AI has been on everyone's lips for many years now. In this section, we will review the most important AI concepts and their corresponding Azure services.

Figure 6.14 is a summary of Azure's AI landscape:

Figure 6.14 – AI solutions in Azure

We say this is a summary because Azure Cognitive Services (alone) can be regrouped into about 20 different services. We already talked about most of these services, but this time we will take the AI-specific angle. Let's start with the machine learning and deep learning options.

Understanding machine learning and deep learning

The services depicted in Figure 6.15 are quite close, in terms of capabilities, and they are all intermingled. This makes it very hard to position them for a specific use case:

Figure 6.15 – Machine learning in Azure

The only no-brainer is Azure Cognitive Services (ACS), a full set of pre-built AI capabilities...

Dealing with other data concerns

In this section, we will review some cross-cutting data concerns, no matter whether you make use of traditional or modern data techniques. Figure 6.17 illustrates some transversal needs, whether you are in a traditional, modern, or big data world:

Figure 6.17 – Cross-cutting data concerns

Let's start with search.

Introducing Azure Cognitive Search

Azure ships the Azure Cognitive Search service. It used to simply be named Azure Search, but it got rebranded to the new name because it also now encompasses other AI capabilities brought by the different cognitive services. Azure Cognitive Search is very well integrated with the other Azure services, which means that you can plug any Azure data store (Cosmos DB, Table Storage, Azure SQL, and so on) into it.

The cognitive part enriches the search contents. For example, the NLP engine will detect entities and extract key phrases and sentiment, while the OCR...

Getting our hands dirty with a near real-time data streaming use case

The time has come to get more acquainted with a few data services. One of the beauties of the cloud is the possibility to ingest and analyze large amounts of data, which are typically involved in big data scenarios. However, big data services can also handle small amounts of data at a very affordable price. To give you a glimpse of what native Azure services can do for you, we will build a quick solution from scratch, and then we'll see it in action.

Here is the scenario:

We have radars located in different cities, and we want to have a real-time dashboard showing both vehicles going over the speed limit and vehicles observing the speed limit (or less).

To make your life easy, we created a small .NET Core console app that simulates the radars. You can download the app from https://github.com/PacktPublishing/The-Azure-Cloud-Native-Architecture-Mapbook/blob/master/Chapter06/code/devicesimulator.zip.

...

Summary

In this chapter, we browsed the vast data landscape of Azure. We split the traditional and modern data technologies to help you understand the possible solutions should you be on the verge of a transition from regular BI to advanced analytics, or from ELT to ETL.

Azure shines in the big data space, and that may be the area where only public cloud providers can make a real difference compared to on-premises systems. We showed you how Azure has both a native big data offering as well as solutions that are open source in origin. Many services are intermingled, but the trend is to let ASA be at the center of everything.

Lastly, we worked on a concrete hands-on exercise to give you a glimpse of what a modern solution might look like. We completed an end-to-end scenario, from data ingestion and real-time queries to surfacing the results on a real-time Power BI dashboard. You should now have a better understanding of when to use what, and how to get started with top-notch data...