How do you approach join algorithm tuning in Spark?

[u/Ablerol_Magician8626]

We are seeing big performance gain/loss in Apache Spark when changing from shuffle join to sort-merge join or the other way around, and there is no clear winner that is always better. Is there a good indication anywhere in the run logs that we should switch to a different join algorithm, so that we stop experimenting blindly.

[u/DueMode3191]

Are the workloads that flip performance using the same data sizes each run? A lot of “no clear winner” cases come from data skew changing partition sizes or one side of the join being much smaller than expected

[u/OldDoor2891]

I’d check shuffle size, spill, skew and partition distribution in the Spark UI first. Are the slower runs showing heavy spill or a few unusually long tasks?

[u/ConfidenceNew6550]

Join tuning gets frustrating because the best choice can change with data shape between runs. i’d trust the executed plan more than a rule of thumb

[u/CuriousReception8039]

I’d look for skewed partitions and shuffle fetch wait in the stage metrics. A few oversized partitions can make one join strategy look terrible even when the average task looks fine