perf: Reduce native shuffle memory overhead by 50%

perf: Reduce native shuffle memory overhead by 50% #4469