In the previous chapter, we discussed how cloud adoption offers greater flexibility and faster deployments for data engineering and analytical workloads. In this chapter, we'll discuss the major tools and services in Microsoft Azure that may help us implement such a solution.
In this chapter, we will cover the following topics:
In recent years, Microsoft Azure has added several powerful services to its arsenal that seamlessly collect, store, process, and publish data for both batch and streaming workloads. Gone are the days where choices for storage and compute were severely limited among cloud vendors. As a user, you simply needed to conform with the supplied tools and services: now, your options are more extensive.
Today, the cloud ecosystem looks very different from what it did previously. The growth of cloud services allows users to choose from a variety of storage, compute, and deployment options. As an example, if I want to run a Spark program, I can choose from at least four different options in Microsoft Azure. The real question is, if all four options are running Apache Spark, then why are these options even required?
Important Note
The array of options available on the cloud are not limited to compute only: the same variety exists for data collection...
Data engineering in Microsoft Azure can be performed using the following three options:
Figure 3.1 – Data engineering option in Microsoft Azure
In the early phases of data engineering, using well-known distributed frameworks such as Hadoop, Spark, and Kafka rose sharply. As a result, many organizations were deploying Hadoop/Spark/Kafka using on-premises infrastructures. Since Hadoop/Spark/Kafka are multi-node frameworks, this meant the installations were performed using physical and virtual machines hosted on either the organization's owned or co-located data centers.
Then came the period when the cloud started to become a reality and organizations started to move their Hadoop/Spark/Kafka clusters to...
In the upcoming chapters, we will be using the services you have just been reading about to build a data lake using the lakehouse architecture. Therefore, it is time to open a free Azure account that gives you 12 months of free services, plus a one-time $260 credit. Please note that not all – but most – services are free. To open a free account and browse through the free services, please visit the following link: https://azure.microsoft.com/en-ca/free/.
Here is some valuable advice:
In this chapter, we learned about the IaaS, PaaS, and SaaS services in Azure that can help a data engineer build a data lake. We also discussed that cloud vendors provide many different options to perform similar operations. It is up to the data engineer to choose the right service that provides the customer with benefits based on their usage patterns, in-house skills, and budget.
The modern-day data pipeline requires careful planning, design, development, and deployment. In the next chapter, we will learn about the life cycle of a data pipeline and effective strategies for each phase of data pipeline creation.
© 2021 Packt Publishing Limited All Rights Reserved
