Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
[SPARK-44021][SQL][FOLLOW-UP] Fix log messages when the number of par…
…tition exceeds maxPartitionNum ### What changes were proposed in this pull request? This PR makes it print the value instead of the option. Previous messages: ``` ./spark.20230707-19.log:23/07/07 19:15:58,858 WARN [HiveServer2-Background-Pool: Thread-121658] datasources.FilePartition:69 : The number of partitions is 298777, which exceeds the maximum number configured: Some(30000). Spark rescales it to 30134 by ignoring the configuration of spark.sql.files.maxPartitionBytes. ./spark.20230707-20.log:23/07/07 20:15:40,911 WARN [HiveServer2-Background-Pool: Thread-130816] datasources.FilePartition:69 : The number of partitions is 175266, which exceeds the maximum number configured: Some(30000). Spark rescales it to 33158 by ignoring the configuration of spark.sql.files.maxPartitionBytes. ./spark.20230707-20.log:23/07/07 20:16:56,748 WARN [HiveServer2-Background-Pool: Thread-131083] datasources.FilePartition:69 : The number of partitions is 298777, which exceeds the maximum number configured: Some(30000). Spark rescales it to 30134 by ignoring the configuration of spark.sql.files.maxPartitionBytes. ``` ### Why are the changes needed? Improve readability. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Manual test. Closes apache#41976 from wangyum/maxPartitionNum. Authored-by: Yuming Wang <[email protected]> Signed-off-by: Yuming Wang <[email protected]>
- Loading branch information