𝗛𝗼𝘄 𝗦𝗽𝗮𝗿𝗸 𝗦𝗵𝘂𝗳𝗳𝗹𝗲 𝗔𝗰𝘁𝘂𝗮𝗹𝗹𝘆 𝗪𝗼𝗿𝗸𝘀

A simple educational dialogue

𝗗𝗮𝘁𝗮 𝗘𝗻𝗴𝗶𝗻𝗲𝗲𝗿:

“My job is slow because of Shuffle. I know it moves data, but how?”

𝗦𝗽𝗮𝗿𝗸:

“It’s a physical relocation process. Let me explain.

It’s not just a network transfer.

It’s a 3-phase process that hits every resource you have: CPU, Disk, Memory, and Network.

Think of it like reorganizing 1TB of data across 100 machines.”

𝗗𝗮𝘁𝗮 𝗘𝗻𝗴𝗶𝗻𝗲𝗲𝗿:

“Walk me through it. What happens first?”

𝗦𝗽𝗮𝗿𝗸:

“Phase 1: Write.

Your executors group data by partition key and write it to local disk.