small shuffle partitions

[javamonkey79]

It's best set the number of shuffle partitions to a small number like one or four in your test suite. This configuration can make your tests run up to 70% faster. You can remove this configuration option or adjust it if you're working with big DataFrames in your test suite.

I have recently found this to be true as well. Do you have any idea why the smaller number of partitions end up being more performant in some cases?

[javamonkey79]

I posted this on stackoverflow here also. It is a fully flushed out example.