Delta Lake

Delta Lake is an evolution in file storage for data tables and merges techniques from data lakes and data warehouse technology. Delta Lake takes the current Parquet technology but adds a transaction log that creates ACID transactions, similar to a database. The key detail to understand is that with transactions, you gain parallelism and reliability, among other features.

Transaction log

One massive adaption Delta Lake gives the traditional data lake is the concept of transactions. A transaction is a small chunk of work that’s accomplished in full. In layman’s terms, when you send data to a Delta table, you can guarantee that data is written as intended without another user creating anything that hinders that process. This avoids dirty reads and writes and inconsistent data. The main component of this feature is the transaction log, which is an ordered log of every transaction that’s made on the table. The transaction log has six actions: adding...