Why Delta Lake Exists and What Plain Parquet Couldn't Do

The four failures that killed the raw data lake

2016. A data engineer kicks off the nightly job that writes 200GB of fresh Parquet into a data lake on object storage. Clean, columnar, cheap to store, queryable by anything. This was the modern architecture, the one that replaced the rigid, expensive data warehouse.

Halfway through, the cluster loses a node and the job dies.

Now the table directory holds…