Chapter 5: Introducing Delta Engine

Delta Engine is the query engine of Delta Lake, which is included by default in Azure Databricks. It is built in a way that allows us to optimize the processing of data in our Delta Lake in a variety of ways, thanks to optimized layouts and improved data indexing. These optimization operations include the use of dynamic file pruning (DFP), Z-Ordering, Auto Compaction, ad hoc processing, and more. The added benefit of these optimization operations is that several of these operations take place in an automatic manner, just by using Delta Lake. You will be using Delta Engine optimization in many ways.

In this chapter, you will learn how to make use of Delta Lake to optimize your Delta Lake ETL in Azure Databricks. Here are the topics on which we will center our discussion:

• Optimizing file management with Delta Engine
• Optimizing queries using DFP
• Using Bloom filters
• Optimizing join performance

Delta Engine is all about optimization...

Technical requirements

To be able to work with Delta Engine in this chapter, you will need an Azure Databricks subscription.

Optimizing file management with Delta Engine

Delta Engine allows improved management of files in Delta Lake, yielding better query speed, thanks to optimization in the layout of the stored data. Delta Lake does this by using two types of algorithms: bin-packing and Z-Ordering. The first algorithm is useful when merging small files into larger ones and is more efficient in handling the larger ones. The second algorithm is borrowed from mathematical analysis and is applied to the underlying structure of the data to map multiple dimensions into one dimension while preserving the locality of the data points.

In this section, we will learn how these algorithms work, see how to implement them using commands that act on our data, and how to handle snapshots of data thanks to the time travel feature.

It is good to remember that although there are automatic optimizations that take place when we use Delta Lake, most of these optimizations do not occur automatically, and some of them must...

Optimizing queries using DFP

DFP is a Delta Lake feature that automatically skips files that are not relevant to a query. It is a default option in Azure Databricks and works by collecting data about files in Delta Lake, without the need to explicitly state that a file should be skipped on a query, improving performance by making use of the granularity of the data.

The behavior of DFP concerning whether a process is enabled or not, the minimum size of a table, and the minimum number of files needed to trigger a process can be managed by the following options:

• spark.databricks.optimizer.dynamicPartitionPruning (default is true): Whether DFP is enabled or not.
• spark.databricks.optimizer.deltaTableSizeThreshold (default is 10 GB): The minimum size of the Delta table that activates DFP.
• spark.databricks.optimizer.deltaTableFilesThreshold (default is 1000): Represents the number of files of the Delta table on the probe side of the join required to trigger DFP. If the...

Using Bloom filters

Bloom filters are a way of efficiently filtering records in a database based on a condition. They have a probabilistic nature and are used to test the membership of an element in a set. We can encounter false positives but not false negatives. These filters were developed as a mathematical construct, to be applied when the amount of data to scan is impractical to be read, and are based on hashing techniques.

Delta Lake provides us with the ability to apply Bloom filters on our queries to further improve performance. We will see how they work at a basic level and how they can be applied in Delta Lake.

Understanding Bloom filters

As mentioned in the introduction to this section, Bloom filters are probabilistic data structures used to test if an element belongs to a category or not. This structure is a fixed-length bit array that is populated using a hash function, which maps the information into ones and zeros. The length of the array depends on the number...

Optimizing join performance

Performing joins on tables can be a resource-expensive operation. To improve the performance of such operations, we can select a subset of the data or correct possible drawbacks, such as having a disproportionate distribution of file sizes in our data. Solving these issues can improve performance and lead to more efficient use of distributed computing power.

Azure Databricks Delta Lake allows optimization of join operations by providing range filtering and correcting skewness in the distribution of the file size of the data in our tables.

Range join optimization

Joins are used frequently, so optimizing these operations can lead to a great improvement in the performance of our queries. Range join optimization is the process of specifying that a join needs to be performed on a subset of data given by a range.

Range join optimization is applied when join operations have a filtering condition whose type is either a numeric or datetime type, and can...

Summary

In this chapter, we have discussed how we can apply Delta Engine features such as Z-Ordering, Data Skipping, and data caching, among others, to improve the layout structure of our data, leading to an overall improvement in query performance. We can leverage these tools to reorganize files into lesser, much more compact ones, in which data is distributed according to how frequently it is accessed. We can also create Bloom filters on data columns to improve the speed in which we run our queries.

In the next chapter, we will deepen our knowledge of how to ingest streams of data in our Delta tables.