What Actually Happens When You Cache a DataFrame

It is lazy, shared, and can be evicted mid-query

Every engineer reaches for cache() when a job runs slow. Most of them make it slower, or change nothing at all, and never find out which.

The problem is that cache() looks like a speed button. You call it, you assume the data is now sitting in memory, and you move on. But nothing about that sentence is guaranteed to be true. cache() is lazy, the storage …