Chapter 5: Deriving Insights from a Data Lakehouse

A lot of ground has been covered so far. The previous chapters covered the methods of ingesting, processing, storing, and serving data in a data lakehouse. Transforming the underlying data into insights is the core aim of any data analytics platform, so this chapter will focus on how to do this. We will also explore the different kinds of data analytics that can be employed in this process.

First, we'll discuss some of the business requirements relating to data analytics. Then, we'll explore different kinds of data analytics and how different stakeholders can use them. After that, we will dive into how these capabilities are enabled using the three components of the data analytics layer: the analytics sandbox, the business intelligence service, and the artificial intelligence service. We will cover different types of descriptive and advanced data analytics. We will also focus on the methods of enabling these analytics...

Discussing the themes of analytics capabilities

Data analytics is the process through which data is transformed into insights. Before getting into the technical components that enable this transformation, let's discuss themes of analytics capabilities that an organization requires. The analytical capabilities are targeted at two types of personas:

• Technical users: Technical users are stakeholders with the technical skills to directly engage with the underlying data structures. They have skills in SQL, programming, data science, and data engineering. A typical technical user could be a data analyst, data engineer, data scientist, or data architect.
• Functional users: Functional users are stakeholders with expertise in the specific functional domain, but their focus on technology is limited. They are subject matter experts, and they understand how the business or a particular domain functions. A typical functional user could be a business analyst or manager.

The...

Enabling analytics capabilities in a data lakehouse

The previous section defined the different types of analytics that need to be fulfilled by a data lakehouse. Now, let's focus on how a data lakehouse enables these capabilities. Recall that in Chapter 2, The Data Lakehouse Architecture Overview, we defined the logical architecture of a data lakehouse. One of the layers of the architecture was the data analytics layer, which interacts with the data lake layer and the data serving layer. The following figure illustrates this interaction between the layers of the data lakehouse architecture:

Figure 5.4 – The interaction between the data lakehouse layers

The three components of the data analytics layer are as follows:

• Analytical sandbox service
• Business intelligence service
• AI/ML service

The following figure maps the required analytics capabilities to the components that fulfill them:

Figure 5.5 –...

Summary

This chapter covered the data analytics layer, which is a vital element in a data lakehouse. We explained that transforming data into insights is the core aim of any data analytics platform, and we looked at how this can be achieved in detail.

We started by exploring some of the different business requirements of data analytics. Then, we covered different users and how they interact with data analytics. We showed how descriptive and advanced analytics form the broad categories of analytics that organizations typically require. We then discussed the characteristics of these categories in detail. Next, we drilled down into different analytics capabilities within these categories. After that, we discussed five analytics capabilities that are important for fulfilling the analytical needs of an organization.

We then mapped the three components of the data analytics layer with their analytics capabilities. Finally, the chapter discussed the sub-components required in each of...

Further reading

For more information regarding the topics that were covered in this chapter, take a look at the following resources:

• Jupyter: https://jupyter.org/
• Machine Learning: https://en.wikipedia.org/wiki/Machine_learning
• Data Science Process: https://becominghuman.ai/data-science-simplified-principles-and-process-b06304d63308
• Deep Learning: https://rpradeepmenon.medium.com/an-executive-primer-to-deep-learning-80c1ece69b34
• Zeppelin: https://zeppelin.apache.org/
• BI Reporting: https://www.jinfonet.com/resources/bi-defined/bi-reporting/
• What is Business Intelligence? https://www.cio.com/article/2439504/business-intelligence-definition-and-solutions.html
• Power B: https://powerbi.microsoft.com/en-us/
• Qlik: https://www.qlik.com/us/
• Tableau: https://www.tableau.com/trial/tableau-online
• Azure Machine Learning: https://azure.microsoft.com/en-us/services/machine-learning/
• Python: https://www.python.org/
• R: https://www.r-project.org...