Working with the OPTIMIZE and ZORDER commands

Delta lake on Databricks lets you speed up queries by changing the layout of the data stored in the cloud storage. The algorithms that support this functionality are as follows:

• Bin-packing: This uses the OPTIMIZE command and helps coalesce small files into larger ones.
• Z-Ordering: This uses the ZORDER command and helps collocate data in the same set of files. This co-locality helps reduce the amount of data that's read by Spark while processing.

Let's learn more about these two layout algorithms with a worked-out example:

1. Run the following code block:

   from pyspark.sql.types import *
   from pyspark.sql.functions import *
   manual_schema = StructType([
     StructField('Year',IntegerType(),True),
     StructField('Month',IntegerType(),True),
     StructField('DayofMonth',IntegerType(),True),
     StructField('DayOfWeek',IntegerType(),True...