Skip to content

[SPARK-44781][SQL] Runtime filter should supports reuse exchange if i… #1500

[SPARK-44781][SQL] Runtime filter should supports reuse exchange if i…

[SPARK-44781][SQL] Runtime filter should supports reuse exchange if i… #1500

Triggered via push August 12, 2023 06:39
Status Failure
Total duration 5h 0m 13s
Artifacts 19

build_main.yml

on: push
Run  /  Check changes
39s
Run / Check changes
Run  /  Base image build
1m 0s
Run / Base image build
Run  /  Breaking change detection with Buf (branch-3.5)
1m 9s
Run / Breaking change detection with Buf (branch-3.5)
Run  /  Scala 2.13 build with SBT
19m 10s
Run / Scala 2.13 build with SBT
Run  /  Run TPC-DS queries with SF=1
1h 7m
Run / Run TPC-DS queries with SF=1
Run  /  Run Docker integration tests
51m 17s
Run / Run Docker integration tests
Run  /  Run Spark on Kubernetes Integration test
1h 7m
Run / Run Spark on Kubernetes Integration test
Matrix: Run / build
Matrix: Run / java-11-17
Run  /  Build modules: sparkr
36m 0s
Run / Build modules: sparkr
Run  /  Linters, licenses, dependencies and documentation generation
9m 28s
Run / Linters, licenses, dependencies and documentation generation
Matrix: Run / pyspark
Fit to window
Zoom out
Zoom in

Annotations

34 errors and 2 warnings
Run / Linters, licenses, dependencies and documentation generation
Process completed with exit code 1.
Run / Run Spark on Kubernetes Integration test
Set() did not contain "decomtest-35ea9189e9256ba9-exec-1".
Run / Run Spark on Kubernetes Integration test
Set() did not contain "decomtest-cf78ba89e92674be-exec-1".
Run / Run Spark on Kubernetes Integration test
sleep interrupted
Run / Run Spark on Kubernetes Integration test
Task io.fabric8.kubernetes.client.utils.internal.SerialExecutor$$Lambda$577/1926024969@b38e86e rejected from java.util.concurrent.ThreadPoolExecutor@4e5fe79[Shutting down, pool size = 3, active threads = 2, queued tasks = 0, completed tasks = 314]
Run / Run Spark on Kubernetes Integration test
Task io.fabric8.kubernetes.client.utils.internal.SerialExecutor$$Lambda$577/1926024969@26380f69 rejected from java.util.concurrent.ThreadPoolExecutor@4e5fe79[Shutting down, pool size = 2, active threads = 1, queued tasks = 0, completed tasks = 315]
Run / Run Spark on Kubernetes Integration test
Set() did not contain "decomtest-49448a89e93b8adb-exec-1".
Run / Run Spark on Kubernetes Integration test
Set() did not contain "decomtest-e2d7d989e93c9b4c-exec-1".
Run / Run Spark on Kubernetes Integration test
Set() did not contain "decomtest-d587d389e94083ee-exec-1".
Run / Run Spark on Kubernetes Integration test
Status(apiVersion=v1, code=404, details=StatusDetails(causes=[], group=null, kind=pods, name=spark-test-app-5e69a45f81734ca48cb9277a8a74f3cc-driver, retryAfterSeconds=null, uid=null, additionalProperties={}), kind=Status, message=pods "spark-test-app-5e69a45f81734ca48cb9277a8a74f3cc-driver" not found, metadata=ListMeta(_continue=null, remainingItemCount=null, resourceVersion=null, selfLink=null, additionalProperties={}), reason=NotFound, status=Failure, additionalProperties={})..
Run / Build modules: sql - extended tests
Process completed with exit code 18.
Run / Build modules: sql - slow tests
File file:/home/runner/work/spark/spark/target/tmp/spark-94e942a7-4fa9-43fa-982f-e2c811d830a8/state/0/2 does not exist
Run / Build modules: sql - slow tests
File file:/home/runner/work/spark/spark/target/tmp/spark-174d2ccd-a784-4f9c-a35a-d740241e3be7/state/0/4 does not exist
Run / Build modules: sql - slow tests
File file:/home/runner/work/spark/spark/target/tmp/spark-cf08899b-2762-4fc6-95e2-00221569049e/state/0/1 does not exist
Run / Build modules: sql - slow tests
File file:/home/runner/work/spark/spark/target/tmp/spark-174d2ccd-a784-4f9c-a35a-d740241e3be7/state/0/0 does not exist
Run / Build modules: sql - slow tests
File file:/home/runner/work/spark/spark/target/tmp/spark-9c3a3122-d3c1-48c5-91fe-a3aef4644eae/state/0/2 does not exist
Run / Build modules: sql - slow tests
File file:/home/runner/work/spark/spark/target/tmp/spark-9c3a3122-d3c1-48c5-91fe-a3aef4644eae/state/0/1 does not exist
Run / Build modules: sql - slow tests
File file:/home/runner/work/spark/spark/target/tmp/spark-cf08899b-2762-4fc6-95e2-00221569049e/state/0/2 does not exist
Run / Build modules: sql - slow tests
File file:/home/runner/work/spark/spark/target/tmp/spark-174d2ccd-a784-4f9c-a35a-d740241e3be7/state/0/1 does not exist
Run / Build modules: sql - slow tests
File file:/home/runner/work/spark/spark/target/tmp/spark-9c3a3122-d3c1-48c5-91fe-a3aef4644eae/state/0/3 does not exist
Run / Build modules: sql - slow tests
File file:/home/runner/work/spark/spark/target/tmp/spark-cf08899b-2762-4fc6-95e2-00221569049e/state/0/3 does not exist
TPCDSV1_4_PlanStabilityWithStatsSuite.check simplified sf100 (tpcds-v1.4/q14a): TPCDSV1_4_PlanStabilityWithStatsSuite#L290
org.scalatest.exceptions.TestFailedException: Plans did not match: last approved simplified plan: /home/runner/work/spark/spark/sql/core/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q14a.sf100/simplified.txt last approved explain plan: /home/runner/work/spark/spark/sql/core/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q14a.sf100/explain.txt TakeOrderedAndProject [channel,i_brand_id,i_class_id,i_category_id,sum(sales),sum(number_sales)] WholeStageCodegen (116) HashAggregate [channel,i_brand_id,i_class_id,i_category_id,spark_grouping_id,sum,isEmpty,sum] [sum(sales),sum(number_salesL),sum(sales),sum(number_sales),sum,isEmpty,sum] InputAdapter Exchange [channel,i_brand_id,i_class_id,i_category_id,spark_grouping_id] #1 WholeStageCodegen (115) HashAggregate [channel,i_brand_id,i_class_id,i_category_id,spark_grouping_id,sales,number_sales] [sum,isEmpty,sum,sum,isEmpty,sum] Expand [sales,number_sales,channel,i_brand_id,i_class_id,i_category_id] InputAdapter Union WholeStageCodegen (38) Project [sales,number_sales,i_brand_id,i_class_id,i_category_id] Filter [sales] Subquery #3 WholeStageCodegen (8) HashAggregate [sum,count] [avg((cast(quantity as decimal(10,0)) * list_price)),average_sales,sum,count] InputAdapter Exchange #15 WholeStageCodegen (7) HashAggregate [quantity,list_price] [sum,count,sum,count] InputAdapter Union WholeStageCodegen (2) Project [ss_quantity,ss_list_price] BroadcastHashJoin [ss_sold_date_sk,d_date_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.store_sales [ss_quantity,ss_list_price,ss_sold_date_sk] ReusedSubquery [d_date_sk] #2 InputAdapter ReusedExchange [d_date_sk] #8 WholeStageCodegen (4) Project [cs_quantity,cs_list_price] BroadcastHashJoin [cs_sold_date_sk,d_date_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.catalog_sales [cs_quantity,cs_list_price,cs_sold_date_sk] ReusedSubquery [d_date_sk] #2 InputAdapter ReusedExchange [d_date_sk] #8 WholeStageCodegen (6) Project [ws_quantity,ws_list_price] BroadcastHashJoin [ws_sold_date_sk,d_date_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.web_sales [ws_quantity,ws_list_price,ws_sold_date_sk] ReusedSubquery [d_date_sk] #2 InputAdapter ReusedExchange [d_date_sk] #8 HashAggregate [i_brand_id,i_class_id,i_category_id,sum,isEmpty,count] [sum((cast(ss_quantity as decimal(10,0)) * ss_list_price)),count(1),sales,number_sales,sum,isEmpty,count] InputAdapter Exchange [i_brand_id,i_class_id,i_category_id] #2 WholeStageCodegen (37) HashAggregate [i_brand_id,i_class_id,i_category_id,ss_quantity,ss_list_price] [sum,isEmpty,count,sum,isEmpty,count] Project [ss_quantity,ss_list_price,i_brand_id,i_class_id,i_category_id] BroadcastHashJoin [ss_item_sk,i_item_sk] Project [ss_item_sk,ss_quantity,ss_list_price] BroadcastHashJoin [ss_sold_date_sk,d_date_sk] BroadcastHashJoin [ss_item_sk,ss_item_sk] Filter [ss_item_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.store_sales [ss_item_sk,ss_quantity,ss_list_price,ss_sold_date_sk] SubqueryBroadcast [d_date_sk] #1 BroadcastExchange #3 WholeStageCodegen (1) Project [d_date_sk] Filter [d_year,d_moy,d_date_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.date_dim [d_date_sk,d_year,d_moy] InputAdapter BroadcastExchange #4 WholeStageCodegen (17) Project [i_item_sk] BroadcastHashJoin [i_brand_id,i_class_id,i_category_id,brand_id,class_id,category_id] Filter [i_brand_id,i_class_id,i_category_id] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.item [i_item_sk,i_brand_id,i_class_id,i_category_id] InputAdapter BroadcastExchange #5 WholeStageCodegen (16) SortMergeJoin [brand_id,class_id,category_id,i_brand_id,i_class_id,i_category_id] InputAdapter WholeStageCodegen (11) Sort [brand_id,class_id,category_id] InputAdapter Exchange [brand_id,class_id,category_id] #6 WholeStageCodegen (10) HashAggregate [brand_id,class_id,category_id] InputAdapter Exchange [brand_id,class_id,category_id] #7 WholeStageCodegen (9) HashAggregate [brand_id,class_id,category_id] Project [i_brand_id,i_class_id,i_category_id] BroadcastHashJoin [ss_item_sk,i_item_sk] Project [ss_item_sk] BroadcastHashJoin [ss_sold_date_sk,d_date_sk] Filter [ss_item_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.store_sales [ss_item_sk,ss_sold_date_sk] SubqueryBroadcast [d_date_sk] #2 BroadcastExchange #8 WholeStageCodegen (1) Project [d_date_sk] Filter [d_year,d_date_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.date_dim [d_date_sk,d_year] InputAdapter ReusedExchange [d_date_sk] #8 InputAdapter BroadcastExchange #9 WholeStageCodegen (8) SortMergeJoin [i_brand_id,i_class_id,i_category_id,i_brand_id,i_class_id,i_category_id] InputAdapter WholeStageCodegen (3) Sort [i_brand_id,i_class_id,i_category_id] InputAdapter Exchange [i_brand_id,i_class_id,i_category_id] #10 WholeStageCodegen (2) Filter [i_item_sk,i_brand_id,i_class_id,i_category_id] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.item [i_item_sk,i_brand_id,i_class_id,i_category_id] InputAdapter WholeStageCodegen (7) Sort [i_brand_id,i_class_id,i_category_id] InputAdapter Exchange [i_brand_id,i_class_id,i_category_id] #11 WholeStageCodegen (6) Project [i_brand_id,i_class_id,i_category_id] BroadcastHashJoin [cs_item_sk,i_item_sk] Project [cs_item_sk] BroadcastHashJoin [cs_sold_date_sk,d_date_sk] Filter [cs_item_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.catalog_sales [cs_item_sk,cs_sold_date_sk] ReusedSubquery [d_date_sk] #2 InputAdapter ReusedExchange [d_date_sk] #8 InputAdapter BroadcastExchange #12 WholeStageCodegen (5) Filter [i_item_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.item [i_item_sk,i_brand_id,i_class_id,i_category_id] InputAdapter WholeStageCodegen (15) Sort [i_brand_id,i_class_id,i_category_id] InputAdapter Exchange [i_brand_id,i_class_id,i_category_id] #13 WholeStageCodegen (14) Project [i_brand_id,i_class_id,i_category_id] BroadcastHashJoin [ws_item_sk,i_item_sk] Project [ws_item_sk] BroadcastHashJoin [ws_sold_date_sk,d_date_sk] Filter [ws_item_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.web_sales [ws_item_sk,ws_sold_date_sk] ReusedSubquery [d_date_sk] #2 InputAdapter ReusedExchange [d_date_sk] #8 InputAdapter ReusedExchange [i_item_sk,i_brand_id,i_class_id,i_category_id] #12 InputAdapter ReusedExchange [d_date_sk] #3 InputAdapter BroadcastExchange #14 WholeStageCodegen (36) BroadcastHashJoin [i_item_sk,ss_item_sk] Filter [i_item_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.item [i_item_sk,i_brand_id,i_class_id,i_category_id] InputAdapter ReusedExchange [ss_item_sk] #4 WholeStageCodegen (76) Project [sales,number_sales,i_brand_id,i_class_id,i_category_id] Filter [sales] ReusedSubquery [average_sales] #3 HashAggregate [i_brand_id,i_class_id,i_category_id,sum,isEmpty,count] [sum((cast(cs_quantity as decimal(10,0)) * cs_list_price)),count(1),sales,number_sales,sum,isEmpty,count] InputAdapter Exchange [i_brand_id,i_class_id,i_category_id] #16 WholeStageCodegen (75) HashAggregate [i_brand_id,i_class_id,i_category_id,cs_quantity,cs_list_price] [sum,isEmpty,count,sum,isEmpty,count] Project [cs_quantity,cs_list_price,i_brand_id,i_class_id,i_category_id] BroadcastHashJoin [cs_item_sk,i_item_sk] Project [cs_item_sk,cs_quantity,cs_list_price] BroadcastHashJoin [cs_sold_date_sk,d_date_sk] BroadcastHashJoin [cs_item_sk,ss_item_sk] Filter [cs_item_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.catalog_sales [cs_item_sk,cs_quantity,cs_list_price,cs_sold_date_sk] ReusedSubquery [d_date_sk] #1 InputAdapter ReusedExchange [ss_item_sk] #4 InputAdapter ReusedExchange [d_date_sk] #3 InputAdapter ReusedExchange [i_item_sk,i_brand_id,i_class_id,i_category_id] #14 WholeStageCodegen (114) Project [sales,number_sales,i_brand_id,i_class_id,i_category_id] Filter [sales] ReusedSubquery [average_sales] #3 HashAggregate [i_brand_id,i_class_id,i_category_id,sum,isEmpty,count] [sum((cast(ws_quantity as decimal(10,0)) * ws_list_price)),count(1),sales,number_sales,sum,isEmpty,count] InputAdapter Exchange [i_brand_id,i_class_id,i_category_id] #17 WholeStageCodegen (113) HashAggregate [i_brand_id,i_class_id,i_category_id,ws_quantity,ws_list_price] [sum,isEmpty,count,sum,isEmpty,count] Project [ws_quantity,ws_list_price,i_brand_id,i_class_id,i_category_id] BroadcastHashJoin [ws_item_sk,i_item_sk] Project [ws_item_sk,ws_quantity,ws_list_price] BroadcastHashJoin [ws_sold_date_sk,d_date_sk] BroadcastHashJoin [ws_item_sk,ss_item_sk] Filter [ws_item_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.web_sales [ws_item_sk,ws_quantity,ws_list_price,ws_sold_date_sk] ReusedSubquery [d_date_sk] #1 InputAdapter ReusedExchange [ss_item_sk] #4 InputAdapter ReusedExchange [d_date_sk] #3 InputAdapter ReusedExchange [i_item_sk,i_brand_id,i_class_id,i_category_id] #14 actual simplified plan: /home/runner/work/spark/spark/target/tmp/q14a.sf100.actual.simplified.txt actual explain plan: /home/runner/work/spark/spark/target/tmp/q14a.sf100.actual.explain.txt TakeOrderedAndProject [channel,i_brand_id,i_class_id,i_category_id,sum(sales),sum(number_sales)] WholeStageCodegen (116) HashAggregate [channel,i_brand_id,i_class_id,i_category_id,spark_grouping_id,sum,isEmpty,sum] [sum(sales),sum(number_salesL),sum(sales),sum(number_sales),sum,isEmpty,sum] InputAdapter Exchange [channel,i_brand_id,i_class_id,i_category_id,spark_grouping_id] #1 WholeStageCodegen (115) HashAggregate [channel,i_brand_id,i_class_id,i_category_id,spark_grouping_id,sales,number_sales] [sum,isEmpty,sum,sum,isEmpty,sum] Expand [sales,number_sales,channel,i_brand_id,i_class_id,i_category_id] InputAdapter Union WholeStageCodegen (38) Project [sales,number_sales,i_brand_id,i_class_id,i_category_id] Filter [sales] Subquery #4 WholeStageCodegen (8) HashAggregate [sum,count] [avg((cast(quantity as decimal(10,0)) * list_price)),average_sales,sum,count] InputAdapter Exchange #16 WholeStageCodegen (7) HashAggregate [quantity,list_price] [sum,count,sum,count] InputAdapter Union WholeStageCodegen (2) Project [ss_quantity,ss_list_price] BroadcastHashJoin [ss_sold_date_sk,d_date_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.store_sales [ss_quantity,ss_list_price,ss_sold_date_sk] ReusedSubquery [d_date_sk] #2 InputAdapter ReusedExchange [d_date_sk] #8 WholeStageCodegen (4) Project [cs_quantity,cs_list_price] BroadcastHashJoin [cs_sold_date_sk,d_date_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.catalog_sales [cs_quantity,cs_list_price,cs_sold_date_sk] ReusedSubquery [d_date_sk] #2 InputAdapter ReusedExchange [d_date_sk] #8 WholeStageCodegen (6) Project [ws_quantity,ws_list_price] BroadcastHashJoin [ws_sold_date_sk,d_date_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.web_sales [ws_quantity,ws_list_price,ws_sold_date_sk] ReusedSubquery [d_date_sk] #2 InputAdapter ReusedExchange [d_date_sk] #8 HashAggregate [i_brand_id,i_class_id,i_category_id,sum,isEmpty,count] [sum((cast(ss_quantity as decimal(10,0)) * ss_list_price)),count(1),sales,number_sales,sum,isEmpty,count] InputAdapter Exchange [i_brand_id,i_class_id,i_category_id] #2 WholeStageCodegen (37) HashAggregate [i_brand_id,i_class_id,i_category_id,ss_quantity,ss_list_price] [sum,isEmpty,count,sum,isEmpty,count] Project [ss_quantity,ss_list_price,i_brand_id,i_class_id,i_category_id] BroadcastHashJoin [ss_item_sk,i_item_sk] Project [ss_item_sk,ss_quantity,ss_list_price] BroadcastHashJoin [ss_sold_date_sk,d_date_sk] BroadcastHashJoin [ss_item_sk,ss_item_sk] Filter [ss_item_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.store_sales [ss_item_sk,ss_quantity,ss_list_price,ss_sold_date_sk] SubqueryBroadcast [d_date_sk] #1 BroadcastExchange #3 WholeStageCodegen (1) Project [d_date_sk] Filter [d_year,d_moy,d_date_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.date_dim [d_date_sk,d_year,d_moy] InputAdapter BroadcastExchange #4 WholeStageCodegen (17) Project [i_item_sk] BroadcastHashJoin [i_brand_id,i_class_id,i_category_id,brand_id,class_id,category_id] Filter [i_brand_id,i_class_id,i_category_id] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.item [i_item_sk,i_brand_id,i_class_id,i_category_id] InputAdapter BroadcastExchange #5 WholeStageCodegen (16) SortMergeJoin [brand_id,class_id,category_id,i_brand_id,i_class_id,i_category_id] InputAdapter WholeStageCodegen (11) Sort [brand_id,class_id,category_id] InputAdapter Exchange [brand_id,class_id,category_id] #6 WholeStageCodegen (10) HashAggregate [brand_id,class_id,category_id] InputAdapter Exchange [brand_id,class_id,category_id] #7 WholeStageCodegen (9) HashAggregate [brand_id,class_id,category_id] Project [i_brand_id,i_class_id,i_category_id] BroadcastHashJoin [ss_item_sk,i_item_sk] Project [ss_item_sk] BroadcastHashJoin [ss_sold_date_sk,d_date_sk] Filter [ss_item_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.store_sales [ss_item_sk,ss_sold_date_sk] SubqueryBroadcast [d_date_sk] #2 BroadcastExchange #8 WholeStageCodegen (1) Project [d_date_sk] Filter [d_year,d_date_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.date_dim [d_date_sk,d_year] InputAdapter ReusedExchange [d_date_sk] #8 InputAdapter BroadcastExchange #9 WholeStageCodegen (8) SortMergeJoin [i_brand_id,i_class_id,i_category_id,i_brand_id,i_class_id,i_category_id] InputAdapter WholeStageCodegen (3) Sort [i_brand_id,i_class_id,i_category_id] InputAdapter Exchange [i_brand_id,i_class_id,i_category_id] #10 WholeStageCodegen (2) Filter [i_item_sk,i_brand_id,i_class_id,i_category_id] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.item [i_item_sk,i_brand_id,i_class_id,i_category_id] InputAdapter WholeStageCodegen (7) Sort [i_brand_id,i_class_id,i_category_id] InputAdapter Exchange [i_brand_id,i_class_id,i_category_id] #11 WholeStageCodegen (6) Project [i_brand_id,i_class_id,i_category_id] BroadcastHashJoin [cs_item_sk,i_item_sk] Project [cs_item_sk] BroadcastHashJoin [cs_sold_date_sk,d_date_sk] Filter [cs_item_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.catalog_sales [cs_item_sk,cs_sold_date_sk] ReusedSubquery [d_date_sk] #2 InputAdapter ReusedExchange [d_date_sk] #8 InputAdapter BroadcastExchange #12 WholeStageCodegen (5) Filter [i_item_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.item [i_item_sk,i_brand_id,i_class_id,i_category_id] InputAdapter WholeStageCodegen (15) Sort [i_brand_id,i_class_id,i_category_id] InputAdapter Exchange [i_brand_id,i_class_id,i_category_id] #13 WholeStageCodegen (14) Project [i_brand_id,i_class_id,i_category_id] BroadcastHashJoin [ws_item_sk,i_item_sk] Project [ws_item_sk] BroadcastHashJoin [ws_sold_date_sk,d_date_sk] Filter [ws_item_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.web_sales [ws_item_sk,ws_sold_date_sk] ReusedSubquery [d_date_sk] #2 InputAdapter ReusedExchange [d_date_sk] #8 InputAdapter ReusedExchange [i_item_sk,i_brand_id,i_class_id,i_category_id] #12 InputAdapter ReusedExchange [d_date_sk] #3 InputAdapter BroadcastExchange #14 WholeStageCodegen (36) BroadcastHashJoin [i_item_sk,ss_item_sk] Filter [i_item_sk] Subquery #3 ObjectHashAggregate [buf] [bloom_filter_agg(xxhash64(ss_item_sk, 42), 5012524, 36583609, 0, 0),bloomFilter,buf] Exchange #15 ObjectHashAggregate [ss_item_sk] [buf,buf] WholeStageCodegen (19) Project [ss_item_sk] BroadcastHashJoin [ss_sold_date_sk,d_date_sk] Project [ss_item_sk,ss_sold_date_sk] Filter [ss_sold_date_sk] ReusedSubquery [d_date_sk] #1 BroadcastHashJoin [ss_item_sk,ss_item_sk] Filter [ss_item_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.store_sales [ss_item_sk,ss_sold_date_sk] InputAdapter ReusedExchange [ss_item_sk] #4 InputAdapter ReusedExchange [d_date_sk] #3 ColumnarToRow InputAdapter Scan parquet spark_catalog.default.item [i_item_sk,i_brand_id,i_class_id,i_category_id] InputAdapter ReusedExchange [ss_item_sk] #4 WholeStageCodegen (76) Project [sales,number_sales,i_brand_id,i_class_id,i_category_id] Filter [sales] ReusedSubquery [average_sales] #4 HashAggregate [i_brand_id,i_class_id,i_category_id,sum,isEmpty,count] [sum((cast(cs_quantity as decimal(10,0)) * cs_list_price)),count(1),sales,number_sales,sum,isEmpty,count] InputAdapter Exchange [i_brand_id,i_class_id,i_category_id] #17 WholeStageCodegen (75) HashAggregate [i_brand_id,i_class_id,i_category_id,cs_quantity,cs_list_price] [sum,isEmpty,count,sum,isEmpty,count] Project [cs_quantity,cs_list_price,i_brand_id,i_class_id,i_category_id] BroadcastHashJoin [cs_item_sk,i_item_sk] Project [cs_item_sk,cs_quantity,cs_list_price] BroadcastHashJoin [cs_sold_date_sk,d_date_sk] BroadcastHashJoin [cs_item_sk,ss_item_sk] Filter [cs_item_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.catalog_sales [cs_item_sk,cs_quantity,cs_list_price,cs_sold_date_sk] ReusedSubquery [d_date_sk] #1 InputAdapter ReusedExchange [ss_item_sk] #4 InputAdapter ReusedExchange [d_date_sk] #3 InputAdapter BroadcastExchange #18 WholeStageCodegen (74) BroadcastHashJoin [i_item_sk,ss_item_sk] Filter [i_item_sk] Subquery #5 ObjectHashAggregate [buf] [bloom_filter_agg(xxhash64(cs_item_sk, 42), 2517915, 20802585, 0, 0),bloomFilter,buf] Exchange #19 ObjectHashAggregate [cs_item_sk] [buf,buf] WholeStageCodegen (19) Project [cs_item_sk] BroadcastHashJoin [cs_sold_date_sk,d_date_sk] Project [cs_item_sk,cs_sold_date_sk] Filter [cs_sold_date_sk] ReusedSubquery [d_date_sk] #1 BroadcastHashJoin [cs_item_sk,ss_item_sk] Filter [cs_item_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.catalog_sales [cs_item_sk,cs_sold_date_sk] InputAdapter ReusedExchange [ss_item_sk] #4 InputAdapter ReusedExchange [d_date_sk] #3 ColumnarToRow InputAdapter Scan parquet spark_catalog.default.item [i_item_sk,i_brand_id,i_class_id,i_category_id] InputAdapter ReusedExchange [ss_item_sk] #4 WholeStageCodegen (114) Project [sales,number_sales,i_brand_id,i_class_id,i_category_id] Filter [sales] ReusedSubquery [average_sales] #4 HashAggregate [i_brand_id,i_class_id,i_category_id,sum,isEmpty,count] [sum((cast(ws_quantity as decimal(10,0)) * ws_list_price)),count(1),sales,number_sales,sum,isEmpty,count] InputAdapter Exchange [i_brand_id,i_class_id,i_category_id] #20 WholeStageCodegen (113) HashAggregate [i_brand_id,i_class_id,i_category_id,ws_quantity,ws_list_price] [sum,isEmpty,count,sum,isEmpty,count] Project [ws_quantity,ws_list_price,i_brand_id,i_class_id,i_category_id] BroadcastHashJoin [ws_item_sk,i_item_sk] Project [ws_item_sk,ws_quantity,ws_list_price] BroadcastHashJoin [ws_sold_date_sk,d_date_sk] BroadcastHashJoin [ws_item_sk,ss_item_sk] Filter [ws_item_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.web_sales [ws_item_sk,ws_quantity,ws_list_price,ws_sold_date_sk] ReusedSubquery [d_date_sk] #1 InputAdapter ReusedExchange [ss_item_sk] #4 InputAdapter ReusedExchange [d_date_sk] #3 InputAdapter BroadcastExchange #21 WholeStageCodegen (112) BroadcastHashJoin [i_item_sk,ss_item_sk] Filter [i_item_sk] Subquery #6 ObjectHashAggregate [buf] [bloom_filter_agg(xxhash64(ws_item_sk, 42), 1252939, 12171674, 0, 0),bloomFilter,buf] Exchange #22 ObjectHashAggregate [ws_item_sk] [buf,buf] WholeStageCodegen (19) Project [ws_item_sk] BroadcastHashJoin [ws_sold_date_sk,d_date_sk] Project [ws_item_sk,ws_sold_date_sk] Filter [ws_sold_date_sk] ReusedSubquery [d_date_sk] #1 BroadcastHashJoin [ws_item_sk,ss_item_sk] Filter [ws_item_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.web_sales [ws_item_sk,ws_sold_date_sk] InputAdapter ReusedExchange [ss_item_sk] #4 InputAdapter ReusedExchange [d_date_sk] #3 ColumnarToRow InputAdapter Scan parquet spark_catalog.default.item [i_item_sk,i_brand_id,i_class_id,i_category_id] InputAdapter ReusedExchange [ss_item_sk] #4
TPCDSV1_4_PlanStabilityWithStatsSuite.check simplified sf100 (tpcds-v1.4/q14b): TPCDSV1_4_PlanStabilityWithStatsSuite#L290
org.scalatest.exceptions.TestFailedException: Plans did not match: last approved simplified plan: /home/runner/work/spark/spark/sql/core/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q14b.sf100/simplified.txt last approved explain plan: /home/runner/work/spark/spark/sql/core/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q14b.sf100/explain.txt TakeOrderedAndProject [i_brand_id,i_class_id,i_category_id,channel,sales,number_sales,channel,i_brand_id,i_class_id,i_category_id,sales,number_sales] WholeStageCodegen (76) BroadcastHashJoin [i_brand_id,i_class_id,i_category_id,i_brand_id,i_class_id,i_category_id] Filter [sales] Subquery #4 WholeStageCodegen (8) HashAggregate [sum,count] [avg((cast(quantity as decimal(10,0)) * list_price)),average_sales,sum,count] InputAdapter Exchange #14 WholeStageCodegen (7) HashAggregate [quantity,list_price] [sum,count,sum,count] InputAdapter Union WholeStageCodegen (2) Project [ss_quantity,ss_list_price] BroadcastHashJoin [ss_sold_date_sk,d_date_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.store_sales [ss_quantity,ss_list_price,ss_sold_date_sk] ReusedSubquery [d_date_sk] #3 InputAdapter ReusedExchange [d_date_sk] #7 WholeStageCodegen (4) Project [cs_quantity,cs_list_price] BroadcastHashJoin [cs_sold_date_sk,d_date_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.catalog_sales [cs_quantity,cs_list_price,cs_sold_date_sk] ReusedSubquery [d_date_sk] #3 InputAdapter ReusedExchange [d_date_sk] #7 WholeStageCodegen (6) Project [ws_quantity,ws_list_price] BroadcastHashJoin [ws_sold_date_sk,d_date_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.web_sales [ws_quantity,ws_list_price,ws_sold_date_sk] ReusedSubquery [d_date_sk] #3 InputAdapter ReusedExchange [d_date_sk] #7 HashAggregate [i_brand_id,i_class_id,i_category_id,sum,isEmpty,count] [sum((cast(ss_quantity as decimal(10,0)) * ss_list_price)),count(1),channel,sales,number_sales,sum,isEmpty,count] InputAdapter Exchange [i_brand_id,i_class_id,i_category_id] #1 WholeStageCodegen (37) HashAggregate [i_brand_id,i_class_id,i_category_id,ss_quantity,ss_list_price] [sum,isEmpty,count,sum,isEmpty,count] Project [ss_quantity,ss_list_price,i_brand_id,i_class_id,i_category_id] BroadcastHashJoin [ss_item_sk,i_item_sk] Project [ss_item_sk,ss_quantity,ss_list_price] BroadcastHashJoin [ss_sold_date_sk,d_date_sk] BroadcastHashJoin [ss_item_sk,ss_item_sk] Filter [ss_item_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.store_sales [ss_item_sk,ss_quantity,ss_list_price,ss_sold_date_sk] SubqueryBroadcast [d_date_sk] #1 BroadcastExchange #2 WholeStageCodegen (1) Project [d_date_sk] Filter [d_week_seq,d_date_sk] ReusedSubquery [d_week_seq] #2 ColumnarToRow InputAdapter Scan parquet spark_catalog.default.date_dim [d_date_sk,d_week_seq] Subquery #2 WholeStageCodegen (1) Project [d_week_seq] Filter [d_year,d_moy,d_dom] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.date_dim [d_week_seq,d_year,d_moy,d_dom] InputAdapter BroadcastExchange #3 WholeStageCodegen (17) Project [i_item_sk] BroadcastHashJoin [i_brand_id,i_class_id,i_category_id,brand_id,class_id,category_id] Filter [i_brand_id,i_class_id,i_category_id] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.item [i_item_sk,i_brand_id,i_class_id,i_category_id] InputAdapter BroadcastExchange #4 WholeStageCodegen (16) SortMergeJoin [brand_id,class_id,category_id,i_brand_id,i_class_id,i_category_id] InputAdapter WholeStageCodegen (11) Sort [brand_id,class_id,category_id] InputAdapter Exchange [brand_id,class_id,category_id] #5 WholeStageCodegen (10) HashAggregate [brand_id,class_id,category_id] InputAdapter Exchange [brand_id,class_id,category_id] #6 WholeStageCodegen (9) HashAggregate [brand_id,class_id,category_id] Project [i_brand_id,i_class_id,i_category_id] BroadcastHashJoin [ss_item_sk,i_item_sk] Project [ss_item_sk] BroadcastHashJoin [ss_sold_date_sk,d_date_sk] Filter [ss_item_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.store_sales [ss_item_sk,ss_sold_date_sk] SubqueryBroadcast [d_date_sk] #3 BroadcastExchange #7 WholeStageCodegen (1) Project [d_date_sk] Filter [d_year,d_date_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.date_dim [d_date_sk,d_year] InputAdapter ReusedExchange [d_date_sk] #7 InputAdapter BroadcastExchange #8 WholeStageCodegen (8) SortMergeJoin [i_brand_id,i_class_id,i_category_id,i_brand_id,i_class_id,i_category_id] InputAdapter WholeStageCodegen (3) Sort [i_brand_id,i_class_id,i_category_id] InputAdapter Exchange [i_brand_id,i_class_id,i_category_id] #9 WholeStageCodegen (2) Filter [i_item_sk,i_brand_id,i_class_id,i_category_id] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.item [i_item_sk,i_brand_id,i_class_id,i_category_id] InputAdapter WholeStageCodegen (7) Sort [i_brand_id,i_class_id,i_category_id] InputAdapter Exchange [i_brand_id,i_class_id,i_category_id] #10 WholeStageCodegen (6) Project [i_brand_id,i_class_id,i_category_id] BroadcastHashJoin [cs_item_sk,i_item_sk] Project [cs_item_sk] BroadcastHashJoin [cs_sold_date_sk,d_date_sk] Filter [cs_item_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.catalog_sales [cs_item_sk,cs_sold_date_sk] ReusedSubquery [d_date_sk] #3 InputAdapter ReusedExchange [d_date_sk] #7 InputAdapter BroadcastExchange #11 WholeStageCodegen (5) Filter [i_item_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.item [i_item_sk,i_brand_id,i_class_id,i_category_id] InputAdapter WholeStageCodegen (15) Sort [i_brand_id,i_class_id,i_category_id] InputAdapter Exchange [i_brand_id,i_class_id,i_category_id] #12 WholeStageCodegen (14) Project [i_brand_id,i_class_id,i_category_id] BroadcastHashJoin [ws_item_sk,i_item_sk] Project [ws_item_sk] BroadcastHashJoin [ws_sold_date_sk,d_date_sk] Filter [ws_item_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.web_sales [ws_item_sk,ws_sold_date_sk] ReusedSubquery [d_date_sk] #3 InputAdapter ReusedExchange [d_date_sk] #7 InputAdapter ReusedExchange [i_item_sk,i_brand_id,i_class_id,i_category_id] #11 InputAdapter ReusedExchange [d_date_sk] #2 InputAdapter BroadcastExchange #13 WholeStageCodegen (36) BroadcastHashJoin [i_item_sk,ss_item_sk] Filter [i_item_sk,i_brand_id,i_class_id,i_category_id] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.item [i_item_sk,i_brand_id,i_class_id,i_category_id] InputAdapter ReusedExchange [ss_item_sk] #3 InputAdapter BroadcastExchange #15 WholeStageCodegen (75) Filter [sales] ReusedSubquery [average_sales] #4 HashAggregate [i_brand_id,i_class_id,i_category_id,sum,isEmpty,count] [sum((cast(ss_quantity as decimal(10,0)) * ss_list_price)),count(1),channel,sales,number_sales,sum,isEmpty,count] InputAdapter Exchange [i_brand_id,i_class_id,i_category_id] #16 WholeStageCodegen (74) HashAggregate [i_brand_id,i_class_id,i_category_id,ss_quantity,ss_list_price] [sum,isEmpty,count,sum,isEmpty,count] Project [ss_quantity,ss_list_price,i_brand_id,i_class_id,i_category_id] BroadcastHashJoin [ss_item_sk,i_item_sk] Project [ss_item_sk,ss_quantity,ss_list_price] BroadcastHashJoin [ss_sold_date_sk,d_date_sk] BroadcastHashJoin [ss_item_sk,ss_item_sk] Filter [ss_item_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.store_sales [ss_item_sk,ss_quantity,ss_list_price,ss_sold_date_sk] SubqueryBroadcast [d_date_sk] #5 BroadcastExchange #17 WholeStageCodegen (1) Project [d_date_sk] Filter [d_week_seq,d_date_sk] ReusedSubquery [d_week_seq] #6 ColumnarToRow InputAdapter Scan parquet spark_catalog.default.date_dim [d_date_sk,d_week_seq] Subquery #6 WholeStageCodegen (1) Project [d_week_seq] Filter [d_year,d_moy,d_dom] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.date_dim [d_week_seq,d_year,d_moy,d_dom] InputAdapter ReusedExchange [ss_item_sk] #3 InputAdapter ReusedExchange [d_date_sk] #17 InputAdapter ReusedExchange [i_item_sk,i_brand_id,i_class_id,i_category_id] #13 actual simplified plan: /home/runner/work/spark/spark/target/tmp/q14b.sf100.actual.simplified.txt actual explain plan: /home/runner/work/spark/spark/target/tmp/q14b.sf100.actual.explain.txt TakeOrderedAndProject [i_brand_id,i_class_id,i_category_id,channel,sales,number_sales,channel,i_brand_id,i_class_id,i_category_id,sales,number_sales] WholeStageCodegen (76) BroadcastHashJoin [i_brand_id,i_class_id,i_category_id,i_brand_id,i_class_id,i_category_id] Filter [sales] Subquery #5 WholeStageCodegen (8) HashAggregate [sum,count] [avg((cast(quantity as decimal(10,0)) * list_price)),average_sales,sum,count] InputAdapter Exchange #15 WholeStageCodegen (7) HashAggregate [quantity,list_price] [sum,count,sum,count] InputAdapter Union WholeStageCodegen (2) Project [ss_quantity,ss_list_price] BroadcastHashJoin [ss_sold_date_sk,d_date_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.store_sales [ss_quantity,ss_list_price,ss_sold_date_sk] ReusedSubquery [d_date_sk] #3 InputAdapter ReusedExchange [d_date_sk] #7 WholeStageCodegen (4) Project [cs_quantity,cs_list_price] BroadcastHashJoin [cs_sold_date_sk,d_date_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.catalog_sales [cs_quantity,cs_list_price,cs_sold_date_sk] ReusedSubquery [d_date_sk] #3 InputAdapter ReusedExchange [d_date_sk] #7 WholeStageCodegen (6) Project [ws_quantity,ws_list_price] BroadcastHashJoin [ws_sold_date_sk,d_date_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.web_sales [ws_quantity,ws_list_price,ws_sold_date_sk] ReusedSubquery [d_date_sk] #3 InputAdapter ReusedExchange [d_date_sk] #7 HashAggregate [i_brand_id,i_class_id,i_category_id,sum,isEmpty,count] [sum((cast(ss_quantity as decimal(10,0)) * ss_list_price)),count(1),channel,sales,number_sales,sum,isEmpty,count] InputAdapter Exchange [i_brand_id,i_class_id,i_category_id] #1 WholeStageCodegen (37) HashAggregate [i_brand_id,i_class_id,i_category_id,ss_quantity,ss_list_price] [sum,isEmpty,count,sum,isEmpty,count] Project [ss_quantity,ss_list_price,i_brand_id,i_class_id,i_category_id] BroadcastHashJoin [ss_item_sk,i_item_sk] Project [ss_item_sk,ss_quantity,ss_list_price] BroadcastHashJoin [ss_sold_date_sk,d_date_sk] BroadcastHashJoin [ss_item_sk,ss_item_sk] Filter [ss_item_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.store_sales [ss_item_sk,ss_quantity,ss_list_price,ss_sold_date_sk] SubqueryBroadcast [d_date_sk] #1 BroadcastExchange #2 WholeStageCodegen (1) Project [d_date_sk] Filter [d_week_seq,d_date_sk] ReusedSubquery [d_week_seq] #2 ColumnarToRow InputAdapter Scan parquet spark_catalog.default.date_dim [d_date_sk,d_week_seq] Subquery #2 WholeStageCodegen (1) Project [d_week_seq] Filter [d_year,d_moy,d_dom] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.date_dim [d_week_seq,d_year,d_moy,d_dom] InputAdapter BroadcastExchange #3 WholeStageCodegen (17) Project [i_item_sk] BroadcastHashJoin [i_brand_id,i_class_id,i_category_id,brand_id,class_id,category_id] Filter [i_brand_id,i_class_id,i_category_id] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.item [i_item_sk,i_brand_id,i_class_id,i_category_id] InputAdapter BroadcastExchange #4 WholeStageCodegen (16) SortMergeJoin [brand_id,class_id,category_id,i_brand_id,i_class_id,i_category_id] InputAdapter WholeStageCodegen (11) Sort [brand_id,class_id,category_id] InputAdapter Exchange [brand_id,class_id,category_id] #5 WholeStageCodegen (10) HashAggregate [brand_id,class_id,category_id] InputAdapter Exchange [brand_id,class_id,category_id] #6 WholeStageCodegen (9) HashAggregate [brand_id,class_id,category_id] Project [i_brand_id,i_class_id,i_category_id] BroadcastHashJoin [ss_item_sk,i_item_sk] Project [ss_item_sk] BroadcastHashJoin [ss_sold_date_sk,d_date_sk] Filter [ss_item_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.store_sales [ss_item_sk,ss_sold_date_sk] SubqueryBroadcast [d_date_sk] #3 BroadcastExchange #7 WholeStageCodegen (1) Project [d_date_sk] Filter [d_year,d_date_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.date_dim [d_date_sk,d_year] InputAdapter ReusedExchange [d_date_sk] #7 InputAdapter BroadcastExchange #8 WholeStageCodegen (8) SortMergeJoin [i_brand_id,i_class_id,i_category_id,i_brand_id,i_class_id,i_category_id] InputAdapter WholeStageCodegen (3) Sort [i_brand_id,i_class_id,i_category_id] InputAdapter Exchange [i_brand_id,i_class_id,i_category_id] #9 WholeStageCodegen (2) Filter [i_item_sk,i_brand_id,i_class_id,i_category_id] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.item [i_item_sk,i_brand_id,i_class_id,i_category_id] InputAdapter WholeStageCodegen (7) Sort [i_brand_id,i_class_id,i_category_id] InputAdapter Exchange [i_brand_id,i_class_id,i_category_id] #10 WholeStageCodegen (6) Project [i_brand_id,i_class_id,i_category_id] BroadcastHashJoin [cs_item_sk,i_item_sk] Project [cs_item_sk] BroadcastHashJoin [cs_sold_date_sk,d_date_sk] Filter [cs_item_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.catalog_sales [cs_item_sk,cs_sold_date_sk] ReusedSubquery [d_date_sk] #3 InputAdapter ReusedExchange [d_date_sk] #7 InputAdapter BroadcastExchange #11 WholeStageCodegen (5) Filter [i_item_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.item [i_item_sk,i_brand_id,i_class_id,i_category_id] InputAdapter WholeStageCodegen (15) Sort [i_brand_id,i_class_id,i_category_id] InputAdapter Exchange [i_brand_id,i_class_id,i_category_id] #12 WholeStageCodegen (14) Project [i_brand_id,i_class_id,i_category_id] BroadcastHashJoin [ws_item_sk,i_item_sk] Project [ws_item_sk] BroadcastHashJoin [ws_sold_date_sk,d_date_sk] Filter [ws_item_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.web_sales [ws_item_sk,ws_sold_date_sk] ReusedSubquery [d_date_sk] #3 InputAdapter ReusedExchange [d_date_sk] #7 InputAdapter ReusedExchange [i_item_sk,i_brand_id,i_class_id,i_category_id] #11 InputAdapter ReusedExchange [d_date_sk] #2 InputAdapter BroadcastExchange #13 WholeStageCodegen (36) BroadcastHashJoin [i_item_sk,ss_item_sk] Filter [i_item_sk,i_brand_id,i_class_id,i_category_id] Subquery #4 ObjectHashAggregate [buf] [bloom_filter_agg(xxhash64(ss_item_sk, 42), 275041999, 67108864, 0, 0),bloomFilter,buf] Exchange #14 ObjectHashAggregate [ss_item_sk] [buf,buf] WholeStageCodegen (19) Project [ss_item_sk] BroadcastHashJoin [ss_sold_date_sk,d_date_sk] Project [ss_item_sk,ss_sold_date_sk] Filter [ss_sold_date_sk] ReusedSubquery [d_date_sk] #1 BroadcastHashJoin [ss_item_sk,ss_item_sk] Filter [ss_item_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.store_sales [ss_item_sk,ss_sold_date_sk] InputAdapter ReusedExchange [ss_item_sk] #3 InputAdapter ReusedExchange [d_date_sk] #2 ColumnarToRow InputAdapter Scan parquet spark_catalog.default.item [i_item_sk,i_brand_id,i_class_id,i_category_id] InputAdapter ReusedExchange [ss_item_sk] #3 InputAdapter BroadcastExchange #16 WholeStageCodegen (75) Filter [sales] ReusedSubquery [average_sales] #5 HashAggregate [i_brand_id,i_class_id,i_category_id,sum,isEmpty,count] [sum((cast(ss_quantity as decimal(10,0)) * ss_list_price)),count(1),channel,sales,number_sales,sum,isEmpty,count] InputAdapter Exchange [i_brand_id,i_class_id,i_category_id] #17 WholeStageCodegen (74) HashAggregate [i_brand_id,i_class_id,i_category_id,ss_quantity,ss_list_price] [sum,isEmpty,count,sum,isEmpty,count] Project [ss_quantity,ss_list_price,i_brand_id,i_class_id,i_category_id] BroadcastHashJoin [ss_item_sk,i_item_sk] Project [ss_item_sk,ss_quantity,ss_list_price] BroadcastHashJoin [ss_sold_date_sk,d_date_sk] BroadcastHashJoin [ss_item_sk,ss_item_sk] Filter [ss_item_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.store_sales [ss_item_sk,ss_quantity,ss_list_price,ss_sold_date_sk] SubqueryBroadcast [d_date_sk] #6 BroadcastExchange #18 WholeStageCodegen (1) Project [d_date_sk] Filter [d_week_seq,d_date_sk] ReusedSubquery [d_week_seq] #7 ColumnarToRow InputAdapter Scan parquet spark_catalog.default.date_dim [d_date_sk,d_week_seq] Subquery #7 WholeStageCodegen (1) Project [d_week_seq] Filter [d_year,d_moy,d_dom] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.date_dim [d_week_seq,d_year,d_moy,d_dom] InputAdapter ReusedExchange [ss_item_sk] #3 InputAdapter ReusedExchange [d_date_sk] #18 InputAdapter BroadcastExchange #19 WholeStageCodegen (73) BroadcastHashJoin [i_item_sk,ss_item_sk] Filter [i_item_sk,i_brand_id,i_class_id,i_category_id] Subquery #8 ObjectHashAggregate [buf] [bloom_filter_agg(xxhash64(ss_item_sk, 42), 275041999, 67108864, 0, 0),bloomFilter,buf] Exchange #20 ObjectHashAggregate [ss_item_sk] [buf,buf] WholeStageCodegen (19) Project [ss_item_sk] BroadcastHashJoin [ss_sold_date_sk,d_date_sk] Project [ss_item_sk,ss_sold_date_sk] Filter [ss_sold_date_sk] ReusedSubquery [d_date_sk] #6 BroadcastHashJoin [ss_item_sk,ss_item_sk] Filter [ss_item_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.store_sales [ss_item_sk,ss_sold_date_sk] InputAdapter ReusedExchange [ss_item_sk] #3 InputAdapter ReusedExchange [d_date_sk] #18 ColumnarToRow InputAdapter Scan parquet spark_catalog.default.item [i_item_sk,i_brand_id,i_class_id,i_category_id] InputAdapter ReusedExchange [ss_item_sk] #3
TPCDSV1_4_PlanStabilityWithStatsSuite.check simplified sf100 (tpcds-v1.4/q17): TPCDSV1_4_PlanStabilityWithStatsSuite#L290
org.scalatest.exceptions.TestFailedException: Plans did not match: last approved simplified plan: /home/runner/work/spark/spark/sql/core/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q17.sf100/simplified.txt last approved explain plan: /home/runner/work/spark/spark/sql/core/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q17.sf100/explain.txt TakeOrderedAndProject [i_item_id,i_item_desc,s_state,store_sales_quantitycount,store_sales_quantityave,store_sales_quantitystdev,store_sales_quantitycov,as_store_returns_quantitycount,as_store_returns_quantityave,as_store_returns_quantitystdev,store_returns_quantitycov,catalog_sales_quantitycount,catalog_sales_quantityave,catalog_sales_quantitystdev,catalog_sales_quantitycov] WholeStageCodegen (18) HashAggregate [i_item_id,i_item_desc,s_state,count,sum,count,n,avg,m2,count,sum,count,n,avg,m2,count,sum,count,n,avg,m2] [count(ss_quantity),avg(ss_quantity),stddev_samp(cast(ss_quantity as double)),count(sr_return_quantity),avg(sr_return_quantity),stddev_samp(cast(sr_return_quantity as double)),count(cs_quantity),avg(cs_quantity),stddev_samp(cast(cs_quantity as double)),store_sales_quantitycount,store_sales_quantityave,store_sales_quantitystdev,store_sales_quantitycov,as_store_returns_quantitycount,as_store_returns_quantityave,as_store_returns_quantitystdev,store_returns_quantitycov,catalog_sales_quantitycount,catalog_sales_quantityave,catalog_sales_quantitystdev,catalog_sales_quantitycov,count,sum,count,n,avg,m2,count,sum,count,n,avg,m2,count,sum,count,n,avg,m2] InputAdapter Exchange [i_item_id,i_item_desc,s_state] #1 WholeStageCodegen (17) HashAggregate [i_item_id,i_item_desc,s_state,ss_quantity,sr_return_quantity,cs_quantity] [count,sum,count,n,avg,m2,count,sum,count,n,avg,m2,count,sum,count,n,avg,m2,count,sum,count,n,avg,m2,count,sum,count,n,avg,m2,count,sum,count,n,avg,m2] Project [ss_quantity,sr_return_quantity,cs_quantity,s_state,i_item_id,i_item_desc] SortMergeJoin [sr_customer_sk,sr_item_sk,cs_bill_customer_sk,cs_item_sk] InputAdapter WholeStageCodegen (13) Sort [sr_customer_sk,sr_item_sk] InputAdapter Exchange [sr_customer_sk,sr_item_sk] #2 WholeStageCodegen (12) Project [ss_quantity,s_state,i_item_id,i_item_desc,sr_item_sk,sr_customer_sk,sr_return_quantity] SortMergeJoin [ss_customer_sk,ss_item_sk,ss_ticket_number,sr_customer_sk,sr_item_sk,sr_ticket_number] InputAdapter WholeStageCodegen (8) Sort [ss_customer_sk,ss_item_sk,ss_ticket_number] InputAdapter Exchange [ss_customer_sk,ss_item_sk,ss_ticket_number] #3 WholeStageCodegen (7) Project [ss_item_sk,ss_customer_sk,ss_ticket_number,ss_quantity,s_state,i_item_id,i_item_desc] SortMergeJoin [ss_item_sk,i_item_sk] InputAdapter WholeStageCodegen (4) Sort [ss_item_sk] InputAdapter Exchange [ss_item_sk] #4 WholeStageCodegen (3) Project [ss_item_sk,ss_customer_sk,ss_ticket_number,ss_quantity,s_state] BroadcastHashJoin [ss_store_sk,s_store_sk] Project [ss_item_sk,ss_customer_sk,ss_store_sk,ss_ticket_number,ss_quantity] BroadcastHashJoin [ss_sold_date_sk,d_date_sk] Filter [ss_customer_sk,ss_item_sk,ss_ticket_number,ss_store_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.store_sales [ss_item_sk,ss_customer_sk,ss_store_sk,ss_ticket_number,ss_quantity,ss_sold_date_sk] SubqueryBroadcast [d_date_sk] #1 BroadcastExchange #5 WholeStageCodegen (1) Project [d_date_sk] Filter [d_quarter_name,d_date_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.date_dim [d_date_sk,d_quarter_name] InputAdapter ReusedExchange [d_date_sk] #5 InputAdapter BroadcastExchange #6 WholeStageCodegen (2) Filter [s_store_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.store [s_store_sk,s_state] InputAdapter WholeStageCodegen (6) Sort [i_item_sk] InputAdapter Exchange [i_item_sk] #7 WholeStageCodegen (5) Filter [i_item_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.item [i_item_sk,i_item_id,i_item_desc] InputAdapter WholeStageCodegen (11) Sort [sr_customer_sk,sr_item_sk,sr_ticket_number] InputAdapter Exchange [sr_customer_sk,sr_item_sk,sr_ticket_number] #8 WholeStageCodegen (10) Project [sr_item_sk,sr_customer_sk,sr_ticket_number,sr_return_quantity] BroadcastHashJoin [sr_returned_date_sk,d_date_sk] Filter [sr_customer_sk,sr_item_sk,sr_ticket_number] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.store_returns [sr_item_sk,sr_customer_sk,sr_ticket_number,sr_return_quantity,sr_returned_date_sk] SubqueryBroadcast [d_date_sk] #2 BroadcastExchange #9 WholeStageCodegen (1) Project [d_date_sk] Filter [d_quarter_name,d_date_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.date_dim [d_date_sk,d_quarter_name] InputAdapter ReusedExchange [d_date_sk] #9 InputAdapter WholeStageCodegen (16) Sort [cs_bill_customer_sk,cs_item_sk] InputAdapter Exchange [cs_bill_customer_sk,cs_item_sk] #10 WholeStageCodegen (15) Project [cs_bill_customer_sk,cs_item_sk,cs_quantity] BroadcastHashJoin [cs_sold_date_sk,d_date_sk] Filter [cs_bill_customer_sk,cs_item_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.catalog_sales [cs_bill_customer_sk,cs_item_sk,cs_quantity,cs_sold_date_sk] ReusedSubquery [d_date_sk] #2 InputAdapter ReusedExchange [d_date_sk] #9 actual simplified plan: /home/runner/work/spark/spark/target/tmp/q17.sf100.actual.simplified.txt actual explain plan: /home/runner/work/spark/spark/target/tmp/q17.sf100.actual.explain.txt TakeOrderedAndProject [i_item_id,i_item_desc,s_state,store_sales_quantitycount,store_sales_quantityave,store_sales_quantitystdev,store_sales_quantitycov,as_store_returns_quantitycount,as_store_returns_quantityave,as_store_returns_quantitystdev,store_returns_quantitycov,catalog_sales_quantitycount,catalog_sales_quantityave,catalog_sales_quantitystdev,catalog_sales_quantitycov] WholeStageCodegen (18) HashAggregate [i_item_id,i_item_desc,s_state,count,sum,count,n,avg,m2,count,sum,count,n,avg,m2,count,sum,count,n,avg,m2] [count(ss_quantity),avg(ss_quantity),stddev_samp(cast(ss_quantity as double)),count(sr_return_quantity),avg(sr_return_quantity),stddev_samp(cast(sr_return_quantity as double)),count(cs_quantity),avg(cs_quantity),stddev_samp(cast(cs_quantity as double)),store_sales_quantitycount,store_sales_quantityave,store_sales_quantitystdev,store_sales_quantitycov,as_store_returns_quantitycount,as_store_returns_quantityave,as_store_returns_quantitystdev,store_returns_quantitycov,catalog_sales_quantitycount,catalog_sales_quantityave,catalog_sales_quantitystdev,catalog_sales_quantitycov,count,sum,count,n,avg,m2,count,sum,count,n,avg,m2,count,sum,count,n,avg,m2] InputAdapter Exchange [i_item_id,i_item_desc,s_state] #1 WholeStageCodegen (17) HashAggregate [i_item_id,i_item_desc,s_state,ss_quantity,sr_return_quantity,cs_quantity] [count,sum,count,n,avg,m2,count,sum,count,n,avg,m2,count,sum,count,n,avg,m2,count,sum,count,n,avg,m2,count,sum,count,n,avg,m2,count,sum,count,n,avg,m2] Project [ss_quantity,sr_return_quantity,cs_quantity,s_state,i_item_id,i_item_desc] SortMergeJoin [sr_customer_sk,sr_item_sk,cs_bill_customer_sk,cs_item_sk] InputAdapter WholeStageCodegen (13) Sort [sr_customer_sk,sr_item_sk] InputAdapter Exchange [sr_customer_sk,sr_item_sk] #2 WholeStageCodegen (12) Project [ss_quantity,s_state,i_item_id,i_item_desc,sr_item_sk,sr_customer_sk,sr_return_quantity] SortMergeJoin [ss_customer_sk,ss_item_sk,ss_ticket_number,sr_customer_sk,sr_item_sk,sr_ticket_number] InputAdapter WholeStageCodegen (8) Sort [ss_customer_sk,ss_item_sk,ss_ticket_number] InputAdapter Exchange [ss_customer_sk,ss_item_sk,ss_ticket_number] #3 WholeStageCodegen (7) Project [ss_item_sk,ss_customer_sk,ss_ticket_number,ss_quantity,s_state,i_item_id,i_item_desc] SortMergeJoin [ss_item_sk,i_item_sk] InputAdapter WholeStageCodegen (4) Sort [ss_item_sk] InputAdapter Exchange [ss_item_sk] #4 WholeStageCodegen (3) Project [ss_item_sk,ss_customer_sk,ss_ticket_number,ss_quantity,s_state] BroadcastHashJoin [ss_store_sk,s_store_sk] Project [ss_item_sk,ss_customer_sk,ss_store_sk,ss_ticket_number,ss_quantity] BroadcastHashJoin [ss_sold_date_sk,d_date_sk] Filter [ss_customer_sk,ss_item_sk,ss_ticket_number,ss_store_sk] Subquery #2 ObjectHashAggregate [buf] [bloom_filter_agg(xxhash64(sr_customer_sk, 42), 4065836, 29674263, 0, 0),bloomFilter,buf] Exchange #6 ObjectHashAggregate [sr_customer_sk] [buf,buf] WholeStageCodegen (2) Project [sr_customer_sk] BroadcastHashJoin [sr_returned_date_sk,d_date_sk] Project [sr_customer_sk,sr_returned_date_sk] Filter [sr_customer_sk,sr_item_sk,sr_ticket_number] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.store_returns [sr_item_sk,sr_customer_sk,sr_ticket_number,sr_returned_date_sk] SubqueryBroadcast [d_date_sk] #3 BroadcastExchange #7 WholeStageCodegen (1) Project [d_date_sk] Filter [d_quarter_name,d_date_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.date_dim [d_date_sk,d_quarter_name] InputAdapter ReusedExchange [d_date_sk] #7 Subquery #4 ObjectHashAggregate [buf] [bloom_filter_agg(xxhash64(sr_item_sk, 42), 4065836, 29674263, 0, 0),bloomFilter,buf] Exchange #8 ObjectHashAggregate [sr_item_sk] [buf,buf] WholeStageCodegen (2) Project [sr_item_sk] BroadcastHashJoin [sr_returned_date_sk,d_date_sk] Project [sr_item_sk,sr_returned_date_sk] Filter [sr_customer_sk,sr_item_sk,sr_ticket_number] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.store_returns [sr_item_sk,sr_customer_sk,sr_ticket_number,sr_returned_date_sk] ReusedSubquery [d_date_sk] #3 InputAdapter ReusedExchange [d_date_sk] #7 Subquery #5 ObjectHashAggregate [buf] [bloom_filter_agg(xxhash64(sr_ticket_number, 42), 4065836, 29674263, 0, 0),bloomFilter,buf] Exchange #9 ObjectHashAggregate [sr_ticket_number] [buf,buf] WholeStageCodegen (2) Project [sr_ticket_number] BroadcastHashJoin [sr_returned_date_sk,d_date_sk] Project [sr_ticket_number,sr_returned_date_sk] Filter [sr_customer_sk,sr_item_sk,sr_ticket_number] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.store_returns [sr_item_sk,sr_customer_sk,sr_ticket_number,sr_returned_date_sk] ReusedSubquery [d_date_sk] #3 InputAdapter ReusedExchange [d_date_sk] #7 ColumnarToRow InputAdapter Scan parquet spark_catalog.default.store_sales [ss_item_sk,ss_customer_sk,ss_store_sk,ss_ticket_number,ss_quantity,ss_sold_date_sk] SubqueryBroadcast [d_date_sk] #1 BroadcastExchange #5 WholeStageCodegen (1) Project [d_date_sk] Filter [d_quarter_name,d_date_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.date_dim [d_date_sk,d_quarter_name] InputAdapter ReusedExchange [d_date_sk] #5 InputAdapter BroadcastExchange #10 WholeStageCodegen (2) Filter [s_store_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.store [s_store_sk,s_state] InputAdapter WholeStageCodegen (6) Sort [i_item_sk] InputAdapter Exchange [i_item_sk] #11 WholeStageCodegen (5) Filter [i_item_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.item [i_item_sk,i_item_id,i_item_desc] InputAdapter WholeStageCodegen (11) Sort [sr_customer_sk,sr_item_sk,sr_ticket_number] InputAdapter Exchange [sr_customer_sk,sr_item_sk,sr_ticket_number] #12 WholeStageCodegen (10) Project [sr_item_sk,sr_customer_sk,sr_ticket_number,sr_return_quantity] BroadcastHashJoin [sr_returned_date_sk,d_date_sk] Filter [sr_customer_sk,sr_item_sk,sr_ticket_number] Subquery #6 ObjectHashAggregate [buf] [bloom_filter_agg(xxhash64(cs_bill_customer_sk, 42), 23056661, 67108864, 0, 0),bloomFilter,buf] Exchange #13 ObjectHashAggregate [cs_bill_customer_sk] [buf,buf] WholeStageCodegen (2) Project [cs_bill_customer_sk] BroadcastHashJoin [cs_sold_date_sk,d_date_sk] Project [cs_bill_customer_sk,cs_sold_date_sk] Filter [cs_bill_customer_sk,cs_item_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.catalog_sales [cs_bill_customer_sk,cs_item_sk,cs_sold_date_sk] ReusedSubquery [d_date_sk] #3 InputAdapter ReusedExchange [d_date_sk] #7 Subquery #7 ObjectHashAggregate [buf] [bloom_filter_agg(xxhash64(cs_item_sk, 42), 23056661, 67108864, 0, 0),bloomFilter,buf] Exchange #14 ObjectHashAggregate [cs_item_sk] [buf,buf] WholeStageCodegen (2) Project [cs_item_sk] BroadcastHashJoin [cs_sold_date_sk,d_date_sk] Project [cs_item_sk,cs_sold_date_sk] Filter [cs_bill_customer_sk,cs_item_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.catalog_sales [cs_bill_customer_sk,cs_item_sk,cs_sold_date_sk] ReusedSubquery [d_date_sk] #3 InputAdapter ReusedExchange [d_date_sk] #7 ColumnarToRow InputAdapter Scan parquet spark_catalog.default.store_returns [sr_item_sk,sr_customer_sk,sr_ticket_number,sr_return_quantity,sr_returned_date_sk] ReusedSubquery [d_date_sk] #3 InputAdapter ReusedExchange [d_date_sk] #7 InputAdapter WholeStageCodegen (16) Sort [cs_bill_customer_sk,cs_item_sk] InputAdapter Exchange [cs_bill_customer_sk,cs_item_sk] #15 WholeStageCodegen (15) Project [cs_bill_customer_sk,cs_item_sk,cs_quantity] BroadcastHashJoin [cs_sold_date_sk,d_date_sk] Filter [cs_bill_customer_sk,cs_item_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.catalog_sales [cs_bill_customer_sk,cs_item_sk,cs_quantity,cs_sold_date_sk] ReusedSubquery [d_date_sk] #3 InputAdapter ReusedExchange [d_date_sk] #7
TPCDSV1_4_PlanStabilityWithStatsSuite.check simplified sf100 (tpcds-v1.4/q23b): TPCDSV1_4_PlanStabilityWithStatsSuite#L290
org.scalatest.exceptions.TestFailedException: Plans did not match: last approved simplified plan: /home/runner/work/spark/spark/sql/core/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q23b.sf100/simplified.txt last approved explain plan: /home/runner/work/spark/spark/sql/core/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q23b.sf100/explain.txt TakeOrderedAndProject [c_last_name,c_first_name,sales] Union WholeStageCodegen (27) HashAggregate [c_last_name,c_first_name,sum,isEmpty] [sum((cast(cs_quantity as decimal(10,0)) * cs_list_price)),sales,sum,isEmpty] InputAdapter Exchange [c_last_name,c_first_name] #1 WholeStageCodegen (26) HashAggregate [c_last_name,c_first_name,cs_quantity,cs_list_price] [sum,isEmpty,sum,isEmpty] Project [cs_quantity,cs_list_price,c_first_name,c_last_name] SortMergeJoin [cs_bill_customer_sk,c_customer_sk] InputAdapter WholeStageCodegen (17) Project [cs_bill_customer_sk,cs_quantity,cs_list_price] BroadcastHashJoin [cs_sold_date_sk,d_date_sk] SortMergeJoin [cs_bill_customer_sk,c_customer_sk] InputAdapter WholeStageCodegen (10) Sort [cs_bill_customer_sk] InputAdapter Exchange [cs_bill_customer_sk] #2 WholeStageCodegen (9) Project [cs_bill_customer_sk,cs_quantity,cs_list_price,cs_sold_date_sk] SortMergeJoin [cs_item_sk,item_sk] InputAdapter WholeStageCodegen (2) Sort [cs_item_sk] InputAdapter Exchange [cs_item_sk] #3 WholeStageCodegen (1) Filter [cs_bill_customer_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.catalog_sales [cs_bill_customer_sk,cs_item_sk,cs_quantity,cs_list_price,cs_sold_date_sk] SubqueryBroadcast [d_date_sk] #1 BroadcastExchange #4 WholeStageCodegen (1) Project [d_date_sk] Filter [d_year,d_moy,d_date_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.date_dim [d_date_sk,d_year,d_moy] InputAdapter WholeStageCodegen (8) Sort [item_sk] Project [item_sk] Filter [cnt] HashAggregate [_groupingexpression,i_item_sk,d_date,count] [count(1),item_sk,cnt,count] HashAggregate [_groupingexpression,i_item_sk,d_date] [count,count] Project [d_date,i_item_sk,i_item_desc] SortMergeJoin [ss_item_sk,i_item_sk] InputAdapter WholeStageCodegen (5) Sort [ss_item_sk] InputAdapter Exchange [ss_item_sk] #5 WholeStageCodegen (4) Project [ss_item_sk,d_date] BroadcastHashJoin [ss_sold_date_sk,d_date_sk] Filter [ss_item_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.store_sales [ss_item_sk,ss_sold_date_sk] SubqueryBroadcast [d_date_sk] #2 BroadcastExchange #6 WholeStageCodegen (1) Project [d_date_sk,d_date] Filter [d_year,d_date_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.date_dim [d_date_sk,d_date,d_year] InputAdapter ReusedExchange [d_date_sk,d_date] #6 InputAdapter WholeStageCodegen (7) Sort [i_item_sk] InputAdapter Exchange [i_item_sk] #7 WholeStageCodegen (6) Filter [i_item_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.item [i_item_sk,i_item_desc] InputAdapter WholeStageCodegen (15) Sort [c_customer_sk] Project [c_customer_sk] Filter [ssales] Subquery #3 WholeStageCodegen (7) HashAggregate [max] [max(csales),tpcds_cmax,max] InputAdapter Exchange #10 WholeStageCodegen (6) HashAggregate [csales] [max,max] HashAggregate [c_customer_sk,sum,isEmpty] [sum((cast(ss_quantity as decimal(10,0)) * ss_sales_price)),csales,sum,isEmpty] HashAggregate [c_customer_sk,ss_quantity,ss_sales_price] [sum,isEmpty,sum,isEmpty] Project [ss_quantity,ss_sales_price,c_customer_sk] SortMergeJoin [ss_customer_sk,c_customer_sk] InputAdapter WholeStageCodegen (3) Sort [ss_customer_sk] InputAdapter Exchange [ss_customer_sk] #11 WholeStageCodegen (2) Project [ss_customer_sk,ss_quantity,ss_sales_price] BroadcastHashJoin [ss_sold_date_sk,d_date_sk] Filter [ss_customer_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.store_sales [ss_customer_sk,ss_quantity,ss_sales_price,ss_sold_date_sk] SubqueryBroadcast [d_date_sk] #4 BroadcastExchange #12 WholeStageCodegen (1) Project [d_date_sk] Filter [d_year,d_date_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.date_dim [d_date_sk,d_year] InputAdapter ReusedExchange [d_date_sk] #12 InputAdapter WholeStageCodegen (5) Sort [c_customer_sk] InputAdapter ReusedExchange [c_customer_sk] #9 HashAggregate [c_customer_sk,sum,isEmpty] [sum((cast(ss_quantity as decimal(10,0)) * ss_sales_price)),ssales,sum,isEmpty] HashAggregate [c_customer_sk,ss_quantity,ss_sales_price] [sum,isEmpty,sum,isEmpty] Project [ss_quantity,ss_sales_price,c_customer_sk] SortMergeJoin [ss_customer_sk,c_customer_sk] InputAdapter WholeStageCodegen (12) Sort [ss_customer_sk] InputAdapter Exchange [ss_customer_sk] #8 WholeStageCodegen (11) Project [ss_customer_sk,ss_quantity,ss_sales_price] Filter [ss_customer_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.store_sales [ss_customer_sk,ss_quantity,ss_sales_price,ss_sold_date_sk] InputAdapter WholeStageCodegen (14) Sort [c_customer_sk] InputAdapter Exchange [c_customer_sk] #9 WholeStageCodegen (13) Filter [c_customer_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.customer [c_customer_sk] InputAdapter ReusedExchange [d_date_sk] #4 InputAdapter WholeStageCodegen (25) SortMergeJoin [c_customer_sk,c_customer_sk] InputAdapter WholeStageCodegen (19) Sort [c_customer_sk] InputAdapter Exchange [c_customer_sk] #13 WholeStageCodegen (18) Filter [c_customer_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.customer [c_customer_sk,c_first_name,c_last_name] InputAdapter WholeStageCodegen (24) Sort [c_customer_sk] Project [c_customer_sk] Filter [ssales] ReusedSubquery [tpcds_cmax] #3 HashAggregate [c_customer_sk,sum,isEmpty] [sum((cast(ss_quantity as decimal(10,0)) * ss_sales_price)),ssales,sum,isEmpty] HashAggregate [c_customer_sk,ss_quantity,ss_sales_price] [sum,isEmpty,sum,isEmpty] Project [ss_quantity,ss_sales_price,c_customer_sk] SortMergeJoin [ss_customer_sk,c_customer_sk] InputAdapter WholeStageCodegen (21) Sort [ss_customer_sk] InputAdapter ReusedExchange [ss_customer_sk,ss_quantity,ss_sales_price] #8 InputAdapter WholeStageCodegen (23) Sort [c_customer_sk] InputAdapter ReusedExchange [c_customer_sk] #9 WholeStageCodegen (54) HashAggregate [c_last_name,c_first_name,sum,isEmpty] [sum((cast(ws_quantity as decimal(10,0)) * ws_list_price)),sales,sum,isEmpty] InputAdapter Exchange [c_last_name,c_first_name] #14 WholeStageCodegen (53) HashAggregate [c_last_name,c_first_name,ws_quantity,ws_list_price] [sum,isEmpty,sum,isEmpty] Project [ws_quantity,ws_list_price,c_first_name,c_last_name] SortMergeJoin [ws_bill_customer_sk,c_customer_sk] InputAdapter WholeStageCodegen (44) Project [ws_bill_customer_sk,ws_quantity,ws_list_price] BroadcastHashJoin [ws_sold_date_sk,d_date_sk] SortMergeJoin [ws_bill_customer_sk,c_customer_sk] InputAdapter WholeStageCodegen (37) Sort [ws_bill_customer_sk] InputAdapter Exchange [ws_bill_customer_sk] #15 WholeStageCodegen (36) Project [ws_bill_customer_sk,ws_quantity,ws_list_price,ws_sold_date_sk] SortMergeJoin [ws_item_sk,item_sk] InputAdapter WholeStageCodegen (29) Sort [ws_item_sk] InputAdapter Exchange [ws_item_sk] #16 WholeStageCodegen (28) Filter [ws_bill_customer_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.web_sales [ws_item_sk,ws_bill_customer_sk,ws_quantity,ws_list_price,ws_sold_date_sk] ReusedSubquery [d_date_sk] #1 InputAdapter WholeStageCodegen (35) Sort [item_sk] Project [item_sk] Filter [cnt] HashAggregate [_groupingexpression,i_item_sk,d_date,count] [count(1),item_sk,cnt,count] HashAggregate [_groupingexpression,i_item_sk,d_date] [count,count] Project [d_date,i_item_sk,i_item_desc] SortMergeJoin [ss_item_sk,i_item_sk] InputAdapter WholeStageCodegen (32) Sort [ss_item_sk] InputAdapter ReusedExchange [ss_item_sk,d_date] #5 InputAdapter WholeStageCodegen (34) Sort [i_item_sk] InputAdapter ReusedExchange [i_item_sk,i_item_desc] #7 InputAdapter WholeStageCodegen (42) Sort [c_customer_sk] Project [c_customer_sk] Filter [ssales] ReusedSubquery [tpcds_cmax] #3 HashAggregate [c_customer_sk,sum,isEmpty] [sum((cast(ss_quantity as decimal(10,0)) * ss_sales_price)),ssales,sum,isEmpty] HashAggregate [c_customer_sk,ss_quantity,ss_sales_price] [sum,isEmpty,sum,isEmpty] Project [ss_quantity,ss_sales_price,c_customer_sk] SortMergeJoin [ss_customer_sk,c_customer_sk] InputAdapter WholeStageCodegen (39) Sort [ss_customer_sk] InputAdapter ReusedExchange [ss_customer_sk,ss_quantity,ss_sales_price] #8 InputAdapter WholeStageCodegen (41) Sort [c_customer_sk] InputAdapter ReusedExchange [c_customer_sk] #9 InputAdapter ReusedExchange [d_date_sk] #4 InputAdapter WholeStageCodegen (52) SortMergeJoin [c_customer_sk,c_customer_sk] InputAdapter WholeStageCodegen (46) Sort [c_customer_sk] InputAdapter ReusedExchange [c_customer_sk,c_first_name,c_last_name] #13 InputAdapter WholeStageCodegen (51) Sort [c_customer_sk] Project [c_customer_sk] Filter [ssales] ReusedSubquery [tpcds_cmax] #3 HashAggregate [c_customer_sk,sum,isEmpty] [sum((cast(ss_quantity as decimal(10,0)) * ss_sales_price)),ssales,sum,isEmpty] HashAggregate [c_customer_sk,ss_quantity,ss_sales_price] [sum,isEmpty,sum,isEmpty] Project [ss_quantity,ss_sales_price,c_customer_sk] SortMergeJoin [ss_customer_sk,c_customer_sk] InputAdapter WholeStageCodegen (48) Sort [ss_customer_sk] InputAdapter ReusedExchange [ss_customer_sk,ss_quantity,ss_sales_price] #8 InputAdapter WholeStageCodegen (50) Sort [c_customer_sk] InputAdapter ReusedExchange [c_customer_sk] #9 actual simplified plan: /home/runner/work/spark/spark/target/tmp/q23b.sf100.actual.simplified.txt actual explain plan: /home/runner/work/spark/spark/target/tmp/q23b.sf100.actual.explain.txt TakeOrderedAndProject [c_last_name,c_first_name,sales] Union WholeStageCodegen (27) HashAggregate [c_last_name,c_first_name,sum,isEmpty] [sum((cast(cs_quantity as decimal(10,0)) * cs_list_price)),sales,sum,isEmpty] InputAdapter Exchange [c_last_name,c_first_name] #1 WholeStageCodegen (26) HashAggregate [c_last_name,c_first_name,cs_quantity,cs_list_price] [sum,isEmpty,sum,isEmpty] Project [cs_quantity,cs_list_price,c_first_name,c_last_name] SortMergeJoin [cs_bill_customer_sk,c_customer_sk] InputAdapter WholeStageCodegen (17) Project [cs_bill_customer_sk,cs_quantity,cs_list_price] BroadcastHashJoin [cs_sold_date_sk,d_date_sk] SortMergeJoin [cs_bill_customer_sk,c_customer_sk] InputAdapter WholeStageCodegen (10) Sort [cs_bill_customer_sk] InputAdapter Exchange [cs_bill_customer_sk] #2 WholeStageCodegen (9) Project [cs_bill_customer_sk,cs_quantity,cs_list_price,cs_sold_date_sk] SortMergeJoin [cs_item_sk,item_sk] InputAdapter WholeStageCodegen (2) Sort [cs_item_sk] InputAdapter Exchange [cs_item_sk] #3 WholeStageCodegen (1) Filter [cs_bill_customer_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.catalog_sales [cs_bill_customer_sk,cs_item_sk,cs_quantity,cs_list_price,cs_sold_date_sk] SubqueryBroadcast [d_date_sk] #1 BroadcastExchange #4 WholeStageCodegen (1) Project [d_date_sk] Filter [d_year,d_moy,d_date_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.date_dim [d_date_sk,d_year,d_moy] InputAdapter WholeStageCodegen (8) Sort [item_sk] Project [item_sk] Filter [cnt] HashAggregate [_groupingexpression,i_item_sk,d_date,count] [count(1),item_sk,cnt,count] HashAggregate [_groupingexpression,i_item_sk,d_date] [count,count] Project [d_date,i_item_sk,i_item_desc] SortMergeJoin [ss_item_sk,i_item_sk] InputAdapter WholeStageCodegen (5) Sort [ss_item_sk] InputAdapter Exchange [ss_item_sk] #5 WholeStageCodegen (4) Project [ss_item_sk,d_date] BroadcastHashJoin [ss_sold_date_sk,d_date_sk] Filter [ss_item_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.store_sales [ss_item_sk,ss_sold_date_sk] SubqueryBroadcast [d_date_sk] #2 BroadcastExchange #6 WholeStageCodegen (1) Project [d_date_sk,d_date] Filter [d_year,d_date_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.date_dim [d_date_sk,d_date,d_year] InputAdapter ReusedExchange [d_date_sk,d_date] #6 InputAdapter WholeStageCodegen (7) Sort [i_item_sk] InputAdapter Exchange [i_item_sk] #7 WholeStageCodegen (6) Filter [i_item_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.item [i_item_sk,i_item_desc] InputAdapter WholeStageCodegen (15) Sort [c_customer_sk] Project [c_customer_sk] Filter [ssales] Subquery #3 WholeStageCodegen (7) HashAggregate [max] [max(csales),tpcds_cmax,max] InputAdapter Exchange #10 WholeStageCodegen (6) HashAggregate [csales] [max,max] HashAggregate [c_customer_sk,sum,isEmpty] [sum((cast(ss_quantity as decimal(10,0)) * ss_sales_price)),csales,sum,isEmpty] HashAggregate [c_customer_sk,ss_quantity,ss_sales_price] [sum,isEmpty,sum,isEmpty] Project [ss_quantity,ss_sales_price,c_customer_sk] SortMergeJoin [ss_customer_sk,c_customer_sk] InputAdapter WholeStageCodegen (3) Sort [ss_customer_sk] InputAdapter Exchange [ss_customer_sk] #11 WholeStageCodegen (2) Project [ss_customer_sk,ss_quantity,ss_sales_price] BroadcastHashJoin [ss_sold_date_sk,d_date_sk] Filter [ss_customer_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.store_sales [ss_customer_sk,ss_quantity,ss_sales_price,ss_sold_date_sk] SubqueryBroadcast [d_date_sk] #4 BroadcastExchange #12 WholeStageCodegen (1) Project [d_date_sk] Filter [d_year,d_date_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.date_dim [d_date_sk,d_year] InputAdapter ReusedExchange [d_date_sk] #12 InputAdapter WholeStageCodegen (5) Sort [c_customer_sk] InputAdapter ReusedExchange [c_customer_sk] #9 HashAggregate [c_customer_sk,sum,isEmpty] [sum((cast(ss_quantity as decimal(10,0)) * ss_sales_price)),ssales,sum,isEmpty] HashAggregate [c_customer_sk,ss_quantity,ss_sales_price] [sum,isEmpty,sum,isEmpty] Project [ss_quantity,ss_sales_price,c_customer_sk] SortMergeJoin [ss_customer_sk,c_customer_sk] InputAdapter WholeStageCodegen (12) Sort [ss_customer_sk] InputAdapter Exchange [ss_customer_sk] #8 WholeStageCodegen (11) Project [ss_customer_sk,ss_quantity,ss_sales_price] Filter [ss_customer_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.store_sales [ss_customer_sk,ss_quantity,ss_sales_price,ss_sold_date_sk] InputAdapter WholeStageCodegen (14) Sort [c_customer_sk] InputAdapter Exchange [c_customer_sk] #9 WholeStageCodegen (13) Filter [c_customer_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.customer [c_customer_sk] InputAdapter ReusedExchange [d_date_sk] #4 InputAdapter WholeStageCodegen (25) SortMergeJoin [c_customer_sk,c_customer_sk] InputAdapter WholeStageCodegen (19) Sort [c_customer_sk] InputAdapter Exchange [c_customer_sk] #13 WholeStageCodegen (18) Filter [c_customer_sk] Subquery #5 ObjectHashAggregate [buf] [bloom_filter_agg(xxhash64(cs_bill_customer_sk, 42), 2516749, 20795378, 0, 0),bloomFilter,buf] Exchange #14 ObjectHashAggregate [cs_bill_customer_sk] [buf,buf] WholeStageCodegen (17) Project [cs_bill_customer_sk] BroadcastHashJoin [cs_sold_date_sk,d_date_sk] Project [cs_bill_customer_sk,cs_sold_date_sk] Filter [cs_sold_date_sk] ReusedSubquery [d_date_sk] #1 SortMergeJoin [cs_bill_customer_sk,c_customer_sk] InputAdapter WholeStageCodegen (10) Sort [cs_bill_customer_sk] InputAdapter Exchange [cs_bill_customer_sk] #15 WholeStageCodegen (9) Project [cs_bill_customer_sk,cs_sold_date_sk] Project [cs_bill_customer_sk,cs_sold_date_sk] SortMergeJoin [cs_item_sk,item_sk] InputAdapter WholeStageCodegen (2) Sort [cs_item_sk] InputAdapter Exchange [cs_item_sk] #16 WholeStageCodegen (1) Filter [cs_bill_customer_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.catalog_sales [cs_bill_customer_sk,cs_item_sk,cs_sold_date_sk] InputAdapter WholeStageCodegen (8) Sort [item_sk] Project [item_sk] Filter [cnt] HashAggregate [_groupingexpression,i_item_sk,d_date,count] [count(1),item_sk,cnt,count] HashAggregate [_groupingexpression,i_item_sk,d_date] [count,count] Project [d_date,i_item_sk,i_item_desc] SortMergeJoin [ss_item_sk,i_item_sk] InputAdapter WholeStageCodegen (5) Sort [ss_item_sk] InputAdapter ReusedExchange [ss_item_sk,d_date] #5 InputAdapter WholeStageCodegen (7) Sort [i_item_sk] InputAdapter ReusedExchange [i_item_sk,i_item_desc] #7 InputAdapter WholeStageCodegen (15) Sort [c_customer_sk] Project [c_customer_sk] Filter [ssales] ReusedSubquery [tpcds_cmax] #3 HashAggregate [c_customer_sk,sum,isEmpty] [sum((cast(ss_quantity as decimal(10,0)) * ss_sales_price)),ssales,sum,isEmpty] HashAggregate [c_customer_sk,ss_quantity,ss_sales_price] [sum,isEmpty,sum,isEmpty] Project [ss_quantity,ss_sales_price,c_customer_sk] SortMergeJoin [ss_customer_sk,c_customer_sk] InputAdapter WholeStageCodegen (12) Sort [ss_customer_sk] InputAdapter ReusedExchange [ss_customer_sk,ss_quantity,ss_sales_price] #8 InputAdapter WholeStageCodegen (14) Sort [c_customer_sk] InputAdapter ReusedExchange [c_customer_sk] #9 InputAdapter ReusedExchange [d_date_sk] #4 ColumnarToRow InputAdapter Scan parquet spark_catalog.default.customer [c_customer_sk,c_first_name,c_last_name] InputAdapter WholeStageCodegen (24) Sort [c_customer_sk] Project [c_customer_sk] Filter [ssales] ReusedSubquery [tpcds_cmax] #3 HashAggregate [c_customer_sk,sum,isEmpty] [sum((cast(ss_quantity as decimal(10,0)) * ss_sales_price)),ssales,sum,isEmpty] HashAggregate [c_customer_sk,ss_quantity,ss_sales_price] [sum,isEmpty,sum,isEmpty] Project [ss_quantity,ss_sales_price,c_customer_sk] SortMergeJoin [ss_customer_sk,c_customer_sk] InputAdapter WholeStageCodegen (21) Sort [ss_customer_sk] InputAdapter ReusedExchange [ss_customer_sk,ss_quantity,ss_sales_price] #8 InputAdapter WholeStageCodegen (23) Sort [c_customer_sk] InputAdapter ReusedExchange [c_customer_sk] #9 WholeStageCodegen (54) HashAggregate [c_last_name,c_first_name,sum,isEmpty] [sum((cast(ws_quantity as decimal(10,0)) * ws_list_price)),sales,sum,isEmpty] InputAdapter Exchange [c_last_name,c_first_name] #17 WholeStageCodegen (53) HashAggregate [c_last_name,c_first_name,ws_quantity,ws_list_price] [sum,isEmpty,sum,isEmpty] Project [ws_quantity,ws_list_price,c_first_name,c_last_name] SortMergeJoin [ws_bill_customer_sk,c_customer_sk] InputAdapter WholeStageCodegen (44) Project [ws_bill_customer_sk,ws_quantity,ws_list_price] BroadcastHashJoin [ws_sold_date_sk,d_date_sk] SortMergeJoin [ws_bill_customer_sk,c_customer_sk] InputAdapter WholeStageCodegen (37) Sort [ws_bill_customer_sk] InputAdapter Exchange [ws_bill_customer_sk] #18 WholeStageCodegen (36) Project [ws_bill_customer_sk,ws_quantity,ws_list_price,ws_sold_date_sk] SortMergeJoin [ws_item_sk,item_sk] InputAdapter WholeStageCodegen (29) Sort [ws_item_sk] InputAdapter Exchange [ws_item_sk] #19 WholeStageCodegen (28) Filter [ws_bill_customer_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.web_sales [ws_item_sk,ws_bill_customer_sk,ws_quantity,ws_list_price,ws_sold_date_sk] ReusedSubquery [d_date_sk] #1 InputAdapter WholeStageCodegen (35) Sort [item_sk] Project [item_sk] Filter [cnt] HashAggregate [_groupingexpression,i_item_sk,d_date,count] [count(1),item_sk,cnt,count] HashAggregate [_groupingexpression,i_item_sk,d_date] [count,count] Project [d_date,i_item_sk,i_item_desc] SortMergeJoin [ss_item_sk,i_item_sk] InputAdapter WholeStageCodegen (32) Sort [ss_item_sk] InputAdapter ReusedExchange [ss_item_sk,d_date] #5 InputAdapter WholeStageCodegen (34) Sort [i_item_sk] InputAdapter ReusedExchange [i_item_sk,i_item_desc] #7 InputAdapter WholeStageCodegen (42) Sort [c_customer_sk] Project [c_customer_sk] Filter [ssales] ReusedSubquery [tpcds_cmax] #3 HashAggregate [c_customer_sk,sum,isEmpty] [sum((cast(ss_quantity as decimal(10,0)) * ss_sales_price)),ssales,sum,isEmpty] HashAggregate [c_customer_sk,ss_quantity,ss_sales_price] [sum,isEmpty,sum,isEmpty] Project [ss_quantity,ss_sales_price,c_customer_sk] SortMergeJoin [ss_customer_sk,c_customer_sk] InputAdapter WholeStageCodegen (39) Sort [ss_customer_sk] InputAdapter ReusedExchange [ss_customer_sk,ss_quantity,ss_sales_price] #8 InputAdapter WholeStageCodegen (41) Sort [c_customer_sk] InputAdapter ReusedExchange [c_customer_sk] #9 InputAdapter ReusedExchange [d_date_sk] #4 InputAdapter WholeStageCodegen (52) SortMergeJoin [c_customer_sk,c_customer_sk] InputAdapter WholeStageCodegen (46) Sort [c_customer_sk] InputAdapter Exchange [c_customer_sk] #20 WholeStageCodegen (45) Filter [c_customer_sk] Subquery #6 ObjectHashAggregate [buf] [bloom_filter_agg(xxhash64(ws_bill_customer_sk, 42), 1252627, 12169292, 0, 0),bloomFilter,buf] Exchange #21 ObjectHashAggregate [ws_bill_customer_sk] [buf,buf] WholeStageCodegen (17) Project [ws_bill_customer_sk] BroadcastHashJoin [ws_sold_date_sk,d_date_sk] Project [ws_bill_customer_sk,ws_sold_date_sk] Filter [ws_sold_date_sk] ReusedSubquery [d_date_sk] #1 SortMergeJoin [ws_bill_customer_sk,c_customer_sk] InputAdapter WholeStageCodegen (10) Sort [ws_bill_customer_sk] InputAdapter Exchange [ws_bill_customer_sk] #22 WholeStageCodegen (9) Project [ws_bill_customer_sk,ws_sold_date_sk] Project [ws_bill_customer_sk,ws_sold_date_sk] SortMergeJoin [ws_item_sk,item_sk] InputAdapter WholeStageCodegen (2) Sort [ws_item_sk] InputAdapter Exchange [ws_item_sk] #23 WholeStageCodegen (1) Filter [ws_bill_customer_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.web_sales [ws_item_sk,ws_bill_customer_sk,ws_sold_date_sk] InputAdapter WholeStageCodegen (8) Sort [item_sk] Project [item_sk] Filter [cnt] HashAggregate [_groupingexpression,i_item_sk,d_date,count] [count(1),item_sk,cnt,count] HashAggregate [_groupingexpression,i_item_sk,d_date] [count,count] Project [d_date,i_item_sk,i_item_desc] SortMergeJoin [ss_item_sk,i_item_sk] InputAdapter WholeStageCodegen (5) Sort [ss_item_sk] InputAdapter ReusedExchange [ss_item_sk,d_date] #5 InputAdapter WholeStageCodegen (7) Sort [i_item_sk] InputAdapter ReusedExchange [i_item_sk,i_item_desc] #7 InputAdapter WholeStageCodegen (15) Sort [c_customer_sk] Project [c_customer_sk] Filter [ssales] ReusedSubquery [tpcds_cmax] #3 HashAggregate [c_customer_sk,sum,isEmpty] [sum((cast(ss_quantity as decimal(10,0)) * ss_sales_price)),ssales,sum,isEmpty] HashAggregate [c_customer_sk,ss_quantity,ss_sales_price] [sum,isEmpty,sum,isEmpty] Project [ss_quantity,ss_sales_price,c_customer_sk] SortMergeJoin [ss_customer_sk,c_customer_sk] InputAdapter WholeStageCodegen (12) Sort [ss_customer_sk] InputAdapter ReusedExchange [ss_customer_sk,ss_quantity,ss_sales_price] #8 InputAdapter WholeStageCodegen (14) Sort [c_customer_sk] InputAdapter ReusedExchange [c_customer_sk] #9 InputAdapter ReusedExchange [d_date_sk] #4 ColumnarToRow InputAdapter Scan parquet spark_catalog.default.customer [c_customer_sk,c_first_name,c_last_name] InputAdapter WholeStageCodegen (51) Sort [c_customer_sk] Project [c_customer_sk] Filter [ssales] ReusedSubquery [tpcds_cmax] #3 HashAggregate [c_customer_sk,sum,isEmpty] [sum((cast(ss_quantity as decimal(10,0)) * ss_sales_price)),ssales,sum,isEmpty] HashAggregate [c_customer_sk,ss_quantity,ss_sales_price] [sum,isEmpty,sum,isEmpty] Project [ss_quantity,ss_sales_price,c_customer_sk] SortMergeJoin [ss_customer_sk,c_customer_sk] InputAdapter WholeStageCodegen (48) Sort [ss_customer_sk] InputAdapter ReusedExchange [ss_customer_sk,ss_quantity,ss_sales_price] #8 InputAdapter WholeStageCodegen (50) Sort [c_customer_sk] InputAdapter ReusedExchange [c_customer_sk] #9
TPCDSV1_4_PlanStabilityWithStatsSuite.check simplified sf100 (tpcds-v1.4/q25): TPCDSV1_4_PlanStabilityWithStatsSuite#L290
org.scalatest.exceptions.TestFailedException: Plans did not match: last approved simplified plan: /home/runner/work/spark/spark/sql/core/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q25.sf100/simplified.txt last approved explain plan: /home/runner/work/spark/spark/sql/core/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q25.sf100/explain.txt TakeOrderedAndProject [i_item_id,i_item_desc,s_store_id,s_store_name,store_sales_profit,store_returns_loss,catalog_sales_profit] WholeStageCodegen (18) HashAggregate [i_item_id,i_item_desc,s_store_id,s_store_name,sum,sum,sum] [sum(UnscaledValue(ss_net_profit)),sum(UnscaledValue(sr_net_loss)),sum(UnscaledValue(cs_net_profit)),store_sales_profit,store_returns_loss,catalog_sales_profit,sum,sum,sum] InputAdapter Exchange [i_item_id,i_item_desc,s_store_id,s_store_name] #1 WholeStageCodegen (17) HashAggregate [i_item_id,i_item_desc,s_store_id,s_store_name,ss_net_profit,sr_net_loss,cs_net_profit] [sum,sum,sum,sum,sum,sum] Project [ss_net_profit,sr_net_loss,cs_net_profit,s_store_id,s_store_name,i_item_id,i_item_desc] SortMergeJoin [sr_customer_sk,sr_item_sk,cs_bill_customer_sk,cs_item_sk] InputAdapter WholeStageCodegen (13) Sort [sr_customer_sk,sr_item_sk] InputAdapter Exchange [sr_customer_sk,sr_item_sk] #2 WholeStageCodegen (12) Project [ss_net_profit,s_store_id,s_store_name,i_item_id,i_item_desc,sr_item_sk,sr_customer_sk,sr_net_loss] SortMergeJoin [ss_customer_sk,ss_item_sk,ss_ticket_number,sr_customer_sk,sr_item_sk,sr_ticket_number] InputAdapter WholeStageCodegen (8) Sort [ss_customer_sk,ss_item_sk,ss_ticket_number] InputAdapter Exchange [ss_customer_sk,ss_item_sk,ss_ticket_number] #3 WholeStageCodegen (7) Project [ss_item_sk,ss_customer_sk,ss_ticket_number,ss_net_profit,s_store_id,s_store_name,i_item_id,i_item_desc] SortMergeJoin [ss_item_sk,i_item_sk] InputAdapter WholeStageCodegen (4) Sort [ss_item_sk] InputAdapter Exchange [ss_item_sk] #4 WholeStageCodegen (3) Project [ss_item_sk,ss_customer_sk,ss_ticket_number,ss_net_profit,s_store_id,s_store_name] BroadcastHashJoin [ss_store_sk,s_store_sk] Project [ss_item_sk,ss_customer_sk,ss_store_sk,ss_ticket_number,ss_net_profit] BroadcastHashJoin [ss_sold_date_sk,d_date_sk] Filter [ss_customer_sk,ss_item_sk,ss_ticket_number,ss_store_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.store_sales [ss_item_sk,ss_customer_sk,ss_store_sk,ss_ticket_number,ss_net_profit,ss_sold_date_sk] SubqueryBroadcast [d_date_sk] #1 BroadcastExchange #5 WholeStageCodegen (1) Project [d_date_sk] Filter [d_moy,d_year,d_date_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.date_dim [d_date_sk,d_year,d_moy] InputAdapter ReusedExchange [d_date_sk] #5 InputAdapter BroadcastExchange #6 WholeStageCodegen (2) Filter [s_store_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.store [s_store_sk,s_store_id,s_store_name] InputAdapter WholeStageCodegen (6) Sort [i_item_sk] InputAdapter Exchange [i_item_sk] #7 WholeStageCodegen (5) Filter [i_item_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.item [i_item_sk,i_item_id,i_item_desc] InputAdapter WholeStageCodegen (11) Sort [sr_customer_sk,sr_item_sk,sr_ticket_number] InputAdapter Exchange [sr_customer_sk,sr_item_sk,sr_ticket_number] #8 WholeStageCodegen (10) Project [sr_item_sk,sr_customer_sk,sr_ticket_number,sr_net_loss] BroadcastHashJoin [sr_returned_date_sk,d_date_sk] Filter [sr_customer_sk,sr_item_sk,sr_ticket_number] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.store_returns [sr_item_sk,sr_customer_sk,sr_ticket_number,sr_net_loss,sr_returned_date_sk] SubqueryBroadcast [d_date_sk] #2 BroadcastExchange #9 WholeStageCodegen (1) Project [d_date_sk] Filter [d_moy,d_year,d_date_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.date_dim [d_date_sk,d_year,d_moy] InputAdapter ReusedExchange [d_date_sk] #9 InputAdapter WholeStageCodegen (16) Sort [cs_bill_customer_sk,cs_item_sk] InputAdapter Exchange [cs_bill_customer_sk,cs_item_sk] #10 WholeStageCodegen (15) Project [cs_bill_customer_sk,cs_item_sk,cs_net_profit] BroadcastHashJoin [cs_sold_date_sk,d_date_sk] Filter [cs_bill_customer_sk,cs_item_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.catalog_sales [cs_bill_customer_sk,cs_item_sk,cs_net_profit,cs_sold_date_sk] ReusedSubquery [d_date_sk] #2 InputAdapter ReusedExchange [d_date_sk] #9 actual simplified plan: /home/runner/work/spark/spark/target/tmp/q25.sf100.actual.simplified.txt actual explain plan: /home/runner/work/spark/spark/target/tmp/q25.sf100.actual.explain.txt TakeOrderedAndProject [i_item_id,i_item_desc,s_store_id,s_store_name,store_sales_profit,store_returns_loss,catalog_sales_profit] WholeStageCodegen (18) HashAggregate [i_item_id,i_item_desc,s_store_id,s_store_name,sum,sum,sum] [sum(UnscaledValue(ss_net_profit)),sum(UnscaledValue(sr_net_loss)),sum(UnscaledValue(cs_net_profit)),store_sales_profit,store_returns_loss,catalog_sales_profit,sum,sum,sum] InputAdapter Exchange [i_item_id,i_item_desc,s_store_id,s_store_name] #1 WholeStageCodegen (17) HashAggregate [i_item_id,i_item_desc,s_store_id,s_store_name,ss_net_profit,sr_net_loss,cs_net_profit] [sum,sum,sum,sum,sum,sum] Project [ss_net_profit,sr_net_loss,cs_net_profit,s_store_id,s_store_name,i_item_id,i_item_desc] SortMergeJoin [sr_customer_sk,sr_item_sk,cs_bill_customer_sk,cs_item_sk] InputAdapter WholeStageCodegen (13) Sort [sr_customer_sk,sr_item_sk] InputAdapter Exchange [sr_customer_sk,sr_item_sk] #2 WholeStageCodegen (12) Project [ss_net_profit,s_store_id,s_store_name,i_item_id,i_item_desc,sr_item_sk,sr_customer_sk,sr_net_loss] SortMergeJoin [ss_customer_sk,ss_item_sk,ss_ticket_number,sr_customer_sk,sr_item_sk,sr_ticket_number] InputAdapter WholeStageCodegen (8) Sort [ss_customer_sk,ss_item_sk,ss_ticket_number] InputAdapter Exchange [ss_customer_sk,ss_item_sk,ss_ticket_number] #3 WholeStageCodegen (7) Project [ss_item_sk,ss_customer_sk,ss_ticket_number,ss_net_profit,s_store_id,s_store_name,i_item_id,i_item_desc] SortMergeJoin [ss_item_sk,i_item_sk] InputAdapter WholeStageCodegen (4) Sort [ss_item_sk] InputAdapter Exchange [ss_item_sk] #4 WholeStageCodegen (3) Project [ss_item_sk,ss_customer_sk,ss_ticket_number,ss_net_profit,s_store_id,s_store_name] BroadcastHashJoin [ss_store_sk,s_store_sk] Project [ss_item_sk,ss_customer_sk,ss_store_sk,ss_ticket_number,ss_net_profit] BroadcastHashJoin [ss_sold_date_sk,d_date_sk] Filter [ss_customer_sk,ss_item_sk,ss_ticket_number,ss_store_sk] Subquery #2 ObjectHashAggregate [buf] [bloom_filter_agg(xxhash64(sr_customer_sk, 42), 2834632, 22720219, 0, 0),bloomFilter,buf] Exchange #6 ObjectHashAggregate [sr_customer_sk] [buf,buf] WholeStageCodegen (2) Project [sr_customer_sk] BroadcastHashJoin [sr_returned_date_sk,d_date_sk] Project [sr_customer_sk,sr_returned_date_sk] Filter [sr_customer_sk,sr_item_sk,sr_ticket_number] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.store_returns [sr_item_sk,sr_customer_sk,sr_ticket_number,sr_returned_date_sk] SubqueryBroadcast [d_date_sk] #3 BroadcastExchange #7 WholeStageCodegen (1) Project [d_date_sk] Filter [d_moy,d_year,d_date_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.date_dim [d_date_sk,d_year,d_moy] InputAdapter ReusedExchange [d_date_sk] #7 Subquery #4 ObjectHashAggregate [buf] [bloom_filter_agg(xxhash64(sr_item_sk, 42), 2834632, 22720219, 0, 0),bloomFilter,buf] Exchange #8 ObjectHashAggregate [sr_item_sk] [buf,buf] WholeStageCodegen (2) Project [sr_item_sk] BroadcastHashJoin [sr_returned_date_sk,d_date_sk] Project [sr_item_sk,sr_returned_date_sk] Filter [sr_customer_sk,sr_item_sk,sr_ticket_number] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.store_returns [sr_item_sk,sr_customer_sk,sr_ticket_number,sr_returned_date_sk] ReusedSubquery [d_date_sk] #3 InputAdapter ReusedExchange [d_date_sk] #7 Subquery #5 ObjectHashAggregate [buf] [bloom_filter_agg(xxhash64(sr_ticket_number, 42), 2834632, 22720219, 0, 0),bloomFilter,buf] Exchange #9 ObjectHashAggregate [sr_ticket_number] [buf,buf] WholeStageCodegen (2) Project [sr_ticket_number] BroadcastHashJoin [sr_returned_date_sk,d_date_sk] Project [sr_ticket_number,sr_returned_date_sk] Filter [sr_customer_sk,sr_item_sk,sr_ticket_number] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.store_returns [sr_item_sk,sr_customer_sk,sr_ticket_number,sr_returned_date_sk] ReusedSubquery [d_date_sk] #3 InputAdapter ReusedExchange [d_date_sk] #7 ColumnarToRow InputAdapter Scan parquet spark_catalog.default.store_sales [ss_item_sk,ss_customer_sk,ss_store_sk,ss_ticket_number,ss_net_profit,ss_sold_date_sk] SubqueryBroadcast [d_date_sk] #1 BroadcastExchange #5 WholeStageCodegen (1) Project [d_date_sk] Filter [d_moy,d_year,d_date_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.date_dim [d_date_sk,d_year,d_moy] InputAdapter ReusedExchange [d_date_sk] #5 InputAdapter BroadcastExchange #10 WholeStageCodegen (2) Filter [s_store_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.store [s_store_sk,s_store_id,s_store_name] InputAdapter WholeStageCodegen (6) Sort [i_item_sk] InputAdapter Exchange [i_item_sk] #11 WholeStageCodegen (5) Filter [i_item_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.item [i_item_sk,i_item_id,i_item_desc] InputAdapter WholeStageCodegen (11) Sort [sr_customer_sk,sr_item_sk,sr_ticket_number] InputAdapter Exchange [sr_customer_sk,sr_item_sk,sr_ticket_number] #12 WholeStageCodegen (10) Project [sr_item_sk,sr_customer_sk,sr_ticket_number,sr_net_loss] BroadcastHashJoin [sr_returned_date_sk,d_date_sk] Filter [sr_customer_sk,sr_item_sk,sr_ticket_number] Subquery #6 ObjectHashAggregate [buf] [bloom_filter_agg(xxhash64(cs_bill_customer_sk, 42), 16074715, 67108864, 0, 0),bloomFilter,buf] Exchange #13 ObjectHashAggregate [cs_bill_customer_sk] [buf,buf] WholeStageCodegen (2) Project [cs_bill_customer_sk] BroadcastHashJoin [cs_sold_date_sk,d_date_sk] Project [cs_bill_customer_sk,cs_sold_date_sk] Filter [cs_bill_customer_sk,cs_item_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.catalog_sales [cs_bill_customer_sk,cs_item_sk,cs_sold_date_sk] ReusedSubquery [d_date_sk] #3 InputAdapter ReusedExchange [d_date_sk] #7 Subquery #7 ObjectHashAggregate [buf] [bloom_filter_agg(xxhash64(cs_item_sk, 42), 16074715, 67108864, 0, 0),bloomFilter,buf] Exchange #14 ObjectHashAggregate [cs_item_sk] [buf,buf] WholeStageCodegen (2) Project [cs_item_sk] BroadcastHashJoin [cs_sold_date_sk,d_date_sk] Project [cs_item_sk,cs_sold_date_sk] Filter [cs_bill_customer_sk,cs_item_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.catalog_sales [cs_bill_customer_sk,cs_item_sk,cs_sold_date_sk] ReusedSubquery [d_date_sk] #3 InputAdapter ReusedExchange [d_date_sk] #7 ColumnarToRow InputAdapter Scan parquet spark_catalog.default.store_returns [sr_item_sk,sr_customer_sk,sr_ticket_number,sr_net_loss,sr_returned_date_sk] ReusedSubquery [d_date_sk] #3 InputAdapter ReusedExchange [d_date_sk] #7 InputAdapter WholeStageCodegen (16) Sort [cs_bill_customer_sk,cs_item_sk] InputAdapter Exchange [cs_bill_customer_sk,cs_item_sk] #15 WholeStageCodegen (15) Project [cs_bill_customer_sk,cs_item_sk,cs_net_profit] BroadcastHashJoin [cs_sold_date_sk,d_date_sk] Filter [cs_bill_customer_sk,cs_item_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.catalog_sales [cs_bill_customer_sk,cs_item_sk,cs_net_profit,cs_sold_date_sk] ReusedSubquery [d_date_sk] #3 InputAdapter ReusedExchange [d_date_sk] #7
TPCDSV1_4_PlanStabilityWithStatsSuite.check simplified sf100 (tpcds-v1.4/q29): TPCDSV1_4_PlanStabilityWithStatsSuite#L290
org.scalatest.exceptions.TestFailedException: Plans did not match: last approved simplified plan: /home/runner/work/spark/spark/sql/core/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q29.sf100/simplified.txt last approved explain plan: /home/runner/work/spark/spark/sql/core/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q29.sf100/explain.txt TakeOrderedAndProject [i_item_id,i_item_desc,s_store_id,s_store_name,store_sales_quantity,store_returns_quantity,catalog_sales_quantity] WholeStageCodegen (18) HashAggregate [i_item_id,i_item_desc,s_store_id,s_store_name,sum,sum,sum] [sum(ss_quantity),sum(sr_return_quantity),sum(cs_quantity),store_sales_quantity,store_returns_quantity,catalog_sales_quantity,sum,sum,sum] InputAdapter Exchange [i_item_id,i_item_desc,s_store_id,s_store_name] #1 WholeStageCodegen (17) HashAggregate [i_item_id,i_item_desc,s_store_id,s_store_name,ss_quantity,sr_return_quantity,cs_quantity] [sum,sum,sum,sum,sum,sum] Project [ss_quantity,sr_return_quantity,cs_quantity,s_store_id,s_store_name,i_item_id,i_item_desc] SortMergeJoin [sr_customer_sk,sr_item_sk,cs_bill_customer_sk,cs_item_sk] InputAdapter WholeStageCodegen (13) Sort [sr_customer_sk,sr_item_sk] InputAdapter Exchange [sr_customer_sk,sr_item_sk] #2 WholeStageCodegen (12) Project [ss_quantity,s_store_id,s_store_name,i_item_id,i_item_desc,sr_item_sk,sr_customer_sk,sr_return_quantity] SortMergeJoin [ss_customer_sk,ss_item_sk,ss_ticket_number,sr_customer_sk,sr_item_sk,sr_ticket_number] InputAdapter WholeStageCodegen (8) Sort [ss_customer_sk,ss_item_sk,ss_ticket_number] InputAdapter Exchange [ss_customer_sk,ss_item_sk,ss_ticket_number] #3 WholeStageCodegen (7) Project [ss_item_sk,ss_customer_sk,ss_ticket_number,ss_quantity,s_store_id,s_store_name,i_item_id,i_item_desc] SortMergeJoin [ss_item_sk,i_item_sk] InputAdapter WholeStageCodegen (4) Sort [ss_item_sk] InputAdapter Exchange [ss_item_sk] #4 WholeStageCodegen (3) Project [ss_item_sk,ss_customer_sk,ss_ticket_number,ss_quantity,s_store_id,s_store_name] BroadcastHashJoin [ss_store_sk,s_store_sk] Project [ss_item_sk,ss_customer_sk,ss_store_sk,ss_ticket_number,ss_quantity] BroadcastHashJoin [ss_sold_date_sk,d_date_sk] Filter [ss_customer_sk,ss_item_sk,ss_ticket_number,ss_store_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.store_sales [ss_item_sk,ss_customer_sk,ss_store_sk,ss_ticket_number,ss_quantity,ss_sold_date_sk] SubqueryBroadcast [d_date_sk] #1 BroadcastExchange #5 WholeStageCodegen (1) Project [d_date_sk] Filter [d_moy,d_year,d_date_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.date_dim [d_date_sk,d_year,d_moy] InputAdapter ReusedExchange [d_date_sk] #5 InputAdapter BroadcastExchange #6 WholeStageCodegen (2) Filter [s_store_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.store [s_store_sk,s_store_id,s_store_name] InputAdapter WholeStageCodegen (6) Sort [i_item_sk] InputAdapter Exchange [i_item_sk] #7 WholeStageCodegen (5) Filter [i_item_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.item [i_item_sk,i_item_id,i_item_desc] InputAdapter WholeStageCodegen (11) Sort [sr_customer_sk,sr_item_sk,sr_ticket_number] InputAdapter Exchange [sr_customer_sk,sr_item_sk,sr_ticket_number] #8 WholeStageCodegen (10) Project [sr_item_sk,sr_customer_sk,sr_ticket_number,sr_return_quantity] BroadcastHashJoin [sr_returned_date_sk,d_date_sk] Filter [sr_customer_sk,sr_item_sk,sr_ticket_number] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.store_returns [sr_item_sk,sr_customer_sk,sr_ticket_number,sr_return_quantity,sr_returned_date_sk] SubqueryBroadcast [d_date_sk] #2 BroadcastExchange #9 WholeStageCodegen (1) Project [d_date_sk] Filter [d_moy,d_year,d_date_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.date_dim [d_date_sk,d_year,d_moy] InputAdapter ReusedExchange [d_date_sk] #9 InputAdapter WholeStageCodegen (16) Sort [cs_bill_customer_sk,cs_item_sk] InputAdapter Exchange [cs_bill_customer_sk,cs_item_sk] #10 WholeStageCodegen (15) Project [cs_bill_customer_sk,cs_item_sk,cs_quantity] BroadcastHashJoin [cs_sold_date_sk,d_date_sk] Filter [cs_bill_customer_sk,cs_item_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.catalog_sales [cs_bill_customer_sk,cs_item_sk,cs_quantity,cs_sold_date_sk] SubqueryBroadcast [d_date_sk] #3 BroadcastExchange #11 WholeStageCodegen (1) Project [d_date_sk] Filter [d_year,d_date_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.date_dim [d_date_sk,d_year] InputAdapter ReusedExchange [d_date_sk] #11 actual simplified plan: /home/runner/work/spark/spark/target/tmp/q29.sf100.actual.simplified.txt actual explain plan: /home/runner/work/spark/spark/target/tmp/q29.sf100.actual.explain.txt TakeOrderedAndProject [i_item_id,i_item_desc,s_store_id,s_store_name,store_sales_quantity,store_returns_quantity,catalog_sales_quantity] WholeStageCodegen (18) HashAggregate [i_item_id,i_item_desc,s_store_id,s_store_name,sum,sum,sum] [sum(ss_quantity),sum(sr_return_quantity),sum(cs_quantity),store_sales_quantity,store_returns_quantity,catalog_sales_quantity,sum,sum,sum] InputAdapter Exchange [i_item_id,i_item_desc,s_store_id,s_store_name] #1 WholeStageCodegen (17) HashAggregate [i_item_id,i_item_desc,s_store_id,s_store_name,ss_quantity,sr_return_quantity,cs_quantity] [sum,sum,sum,sum,sum,sum] Project [ss_quantity,sr_return_quantity,cs_quantity,s_store_id,s_store_name,i_item_id,i_item_desc] SortMergeJoin [sr_customer_sk,sr_item_sk,cs_bill_customer_sk,cs_item_sk] InputAdapter WholeStageCodegen (13) Sort [sr_customer_sk,sr_item_sk] InputAdapter Exchange [sr_customer_sk,sr_item_sk] #2 WholeStageCodegen (12) Project [ss_quantity,s_store_id,s_store_name,i_item_id,i_item_desc,sr_item_sk,sr_customer_sk,sr_return_quantity] SortMergeJoin [ss_customer_sk,ss_item_sk,ss_ticket_number,sr_customer_sk,sr_item_sk,sr_ticket_number] InputAdapter WholeStageCodegen (8) Sort [ss_customer_sk,ss_item_sk,ss_ticket_number] InputAdapter Exchange [ss_customer_sk,ss_item_sk,ss_ticket_number] #3 WholeStageCodegen (7) Project [ss_item_sk,ss_customer_sk,ss_ticket_number,ss_quantity,s_store_id,s_store_name,i_item_id,i_item_desc] SortMergeJoin [ss_item_sk,i_item_sk] InputAdapter WholeStageCodegen (4) Sort [ss_item_sk] InputAdapter Exchange [ss_item_sk] #4 WholeStageCodegen (3) Project [ss_item_sk,ss_customer_sk,ss_ticket_number,ss_quantity,s_store_id,s_store_name] BroadcastHashJoin [ss_store_sk,s_store_sk] Project [ss_item_sk,ss_customer_sk,ss_store_sk,ss_ticket_number,ss_quantity] BroadcastHashJoin [ss_sold_date_sk,d_date_sk] Filter [ss_customer_sk,ss_item_sk,ss_ticket_number,ss_store_sk] Subquery #2 ObjectHashAggregate [buf] [bloom_filter_agg(xxhash64(sr_customer_sk, 42), 1417316, 13404864, 0, 0),bloomFilter,buf] Exchange #6 ObjectHashAggregate [sr_customer_sk] [buf,buf] WholeStageCodegen (2) Project [sr_customer_sk] BroadcastHashJoin [sr_returned_date_sk,d_date_sk] Project [sr_customer_sk,sr_returned_date_sk] Filter [sr_customer_sk,sr_item_sk,sr_ticket_number] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.store_returns [sr_item_sk,sr_customer_sk,sr_ticket_number,sr_returned_date_sk] SubqueryBroadcast [d_date_sk] #3 BroadcastExchange #7 WholeStageCodegen (1) Project [d_date_sk] Filter [d_moy,d_year,d_date_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.date_dim [d_date_sk,d_year,d_moy] InputAdapter ReusedExchange [d_date_sk] #7 Subquery #4 ObjectHashAggregate [buf] [bloom_filter_agg(xxhash64(sr_item_sk, 42), 1417316, 13404864, 0, 0),bloomFilter,buf] Exchange #8 ObjectHashAggregate [sr_item_sk] [buf,buf] WholeStageCodegen (2) Project [sr_item_sk] BroadcastHashJoin [sr_returned_date_sk,d_date_sk] Project [sr_item_sk,sr_returned_date_sk] Filter [sr_customer_sk,sr_item_sk,sr_ticket_number] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.store_returns [sr_item_sk,sr_customer_sk,sr_ticket_number,sr_returned_date_sk] ReusedSubquery [d_date_sk] #3 InputAdapter ReusedExchange [d_date_sk] #7 Subquery #5 ObjectHashAggregate [buf] [bloom_filter_agg(xxhash64(sr_ticket_number, 42), 1417316, 13404864, 0, 0),bloomFilter,buf] Exchange #9 ObjectHashAggregate [sr_ticket_number] [buf,buf] WholeStageCodegen (2) Project [sr_ticket_number] BroadcastHashJoin [sr_returned_date_sk,d_date_sk] Project [sr_ticket_number,sr_returned_date_sk] Filter [sr_customer_sk,sr_item_sk,sr_ticket_number] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.store_returns [sr_item_sk,sr_customer_sk,sr_ticket_number,sr_returned_date_sk] ReusedSubquery [d_date_sk] #3 InputAdapter ReusedExchange [d_date_sk] #7 ColumnarToRow InputAdapter Scan parquet spark_catalog.default.store_sales [ss_item_sk,ss_customer_sk,ss_store_sk,ss_ticket_number,ss_quantity,ss_sold_date_sk] SubqueryBroadcast [d_date_sk] #1 BroadcastExchange #5 WholeStageCodegen (1) Project [d_date_sk] Filter [d_moy,d_year,d_date_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.date_dim [d_date_sk,d_year,d_moy] InputAdapter ReusedExchange [d_date_sk] #5 InputAdapter BroadcastExchange #10 WholeStageCodegen (2) Filter [s_store_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.store [s_store_sk,s_store_id,s_store_name] InputAdapter WholeStageCodegen (6) Sort [i_item_sk] InputAdapter Exchange [i_item_sk] #11 WholeStageCodegen (5) Filter [i_item_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.item [i_item_sk,i_item_id,i_item_desc] InputAdapter WholeStageCodegen (11) Sort [sr_customer_sk,sr_item_sk,sr_ticket_number] InputAdapter Exchange [sr_customer_sk,sr_item_sk,sr_ticket_number] #12 WholeStageCodegen (10) Project [sr_item_sk,sr_customer_sk,sr_ticket_number,sr_return_quantity] BroadcastHashJoin [sr_returned_date_sk,d_date_sk] Filter [sr_customer_sk,sr_item_sk,sr_ticket_number] Subquery #6 ObjectHashAggregate [buf] [bloom_filter_agg(xxhash64(cs_bill_customer_sk, 42), 88086186, 67108864, 0, 0),bloomFilter,buf] Exchange #13 ObjectHashAggregate [cs_bill_customer_sk] [buf,buf] WholeStageCodegen (2) Project [cs_bill_customer_sk] BroadcastHashJoin [cs_sold_date_sk,d_date_sk] Project [cs_bill_customer_sk,cs_sold_date_sk] Filter [cs_bill_customer_sk,cs_item_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.catalog_sales [cs_bill_customer_sk,cs_item_sk,cs_sold_date_sk] SubqueryBroadcast [d_date_sk] #7 BroadcastExchange #14 WholeStageCodegen (1) Project [d_date_sk] Filter [d_year,d_date_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.date_dim [d_date_sk,d_year] InputAdapter ReusedExchange [d_date_sk] #14 Subquery #8 ObjectHashAggregate [buf] [bloom_filter_agg(xxhash64(cs_item_sk, 42), 88086186, 67108864, 0, 0),bloomFilter,buf] Exchange #15 ObjectHashAggregate [cs_item_sk] [buf,buf] WholeStageCodegen (2) Project [cs_item_sk] BroadcastHashJoin [cs_sold_date_sk,d_date_sk] Project [cs_item_sk,cs_sold_date_sk] Filter [cs_bill_customer_sk,cs_item_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.catalog_sales [cs_bill_customer_sk,cs_item_sk,cs_sold_date_sk] ReusedSubquery [d_date_sk] #7 InputAdapter ReusedExchange [d_date_sk] #14 ColumnarToRow InputAdapter Scan parquet spark_catalog.default.store_returns [sr_item_sk,sr_customer_sk,sr_ticket_number,sr_return_quantity,sr_returned_date_sk] ReusedSubquery [d_date_sk] #3 InputAdapter ReusedExchange [d_date_sk] #7 InputAdapter WholeStageCodegen (16) Sort [cs_bill_customer_sk,cs_item_sk] InputAdapter Exchange [cs_bill_customer_sk,cs_item_sk] #16 WholeStageCodegen (15) Project [cs_bill_customer_sk,cs_item_sk,cs_quantity] BroadcastHashJoin [cs_sold_date_sk,d_date_sk] Filter [cs_bill_customer_sk,cs_item_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.catalog_sales [cs_bill_customer_sk,cs_item_sk,cs_quantity,cs_sold_date_sk] ReusedSubquery [d_date_sk] #7 InputAdapter ReusedExchange [d_date_sk] #14
TPCDSV1_4_PlanStabilityWithStatsSuite.check simplified sf100 (tpcds-v1.4/q50): TPCDSV1_4_PlanStabilityWithStatsSuite#L290
org.scalatest.exceptions.TestFailedException: Plans did not match: last approved simplified plan: /home/runner/work/spark/spark/sql/core/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q50.sf100/simplified.txt last approved explain plan: /home/runner/work/spark/spark/sql/core/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q50.sf100/explain.txt TakeOrderedAndProject [s_store_name,s_company_id,s_street_number,s_street_name,s_street_type,s_suite_number,s_city,s_county,s_state,s_zip,30 days ,31 - 60 days ,61 - 90 days ,91 - 120 days ,>120 days ] WholeStageCodegen (9) HashAggregate [s_store_name,s_company_id,s_street_number,s_street_name,s_street_type,s_suite_number,s_city,s_county,s_state,s_zip,sum,sum,sum,sum,sum] [sum(CASE WHEN ((sr_returned_date_sk - ss_sold_date_sk) <= 30) THEN 1 ELSE 0 END),sum(CASE WHEN (((sr_returned_date_sk - ss_sold_date_sk) > 30) AND ((sr_returned_date_sk - ss_sold_date_sk) <= 60)) THEN 1 ELSE 0 END),sum(CASE WHEN (((sr_returned_date_sk - ss_sold_date_sk) > 60) AND ((sr_returned_date_sk - ss_sold_date_sk) <= 90)) THEN 1 ELSE 0 END),sum(CASE WHEN (((sr_returned_date_sk - ss_sold_date_sk) > 90) AND ((sr_returned_date_sk - ss_sold_date_sk) <= 120)) THEN 1 ELSE 0 END),sum(CASE WHEN ((sr_returned_date_sk - ss_sold_date_sk) > 120) THEN 1 ELSE 0 END),30 days ,31 - 60 days ,61 - 90 days ,91 - 120 days ,>120 days ,sum,sum,sum,sum,sum] InputAdapter Exchange [s_store_name,s_company_id,s_street_number,s_street_name,s_street_type,s_suite_number,s_city,s_county,s_state,s_zip] #1 WholeStageCodegen (8) HashAggregate [s_store_name,s_company_id,s_street_number,s_street_name,s_street_type,s_suite_number,s_city,s_county,s_state,s_zip,sr_returned_date_sk,ss_sold_date_sk] [sum,sum,sum,sum,sum,sum,sum,sum,sum,sum] Project [ss_sold_date_sk,sr_returned_date_sk,s_store_name,s_company_id,s_street_number,s_street_name,s_street_type,s_suite_number,s_city,s_county,s_state,s_zip] BroadcastHashJoin [ss_store_sk,s_store_sk] Project [sr_returned_date_sk,ss_store_sk,ss_sold_date_sk] BroadcastHashJoin [ss_sold_date_sk,d_date_sk] Project [sr_returned_date_sk,ss_store_sk,ss_sold_date_sk] SortMergeJoin [sr_ticket_number,sr_item_sk,sr_customer_sk,ss_ticket_number,ss_item_sk,ss_customer_sk] InputAdapter WholeStageCodegen (3) Sort [sr_ticket_number,sr_item_sk,sr_customer_sk] InputAdapter Exchange [sr_ticket_number,sr_item_sk,sr_customer_sk] #2 WholeStageCodegen (2) Project [sr_item_sk,sr_customer_sk,sr_ticket_number,sr_returned_date_sk] BroadcastHashJoin [sr_returned_date_sk,d_date_sk] Filter [sr_ticket_number,sr_item_sk,sr_customer_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.store_returns [sr_item_sk,sr_customer_sk,sr_ticket_number,sr_returned_date_sk] SubqueryBroadcast [d_date_sk] #1 BroadcastExchange #3 WholeStageCodegen (1) Project [d_date_sk] Filter [d_year,d_moy,d_date_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.date_dim [d_date_sk,d_year,d_moy] InputAdapter ReusedExchange [d_date_sk] #3 InputAdapter WholeStageCodegen (5) Sort [ss_ticket_number,ss_item_sk,ss_customer_sk] InputAdapter Exchange [ss_ticket_number,ss_item_sk,ss_customer_sk] #4 WholeStageCodegen (4) Filter [ss_ticket_number,ss_item_sk,ss_customer_sk,ss_store_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.store_sales [ss_item_sk,ss_customer_sk,ss_store_sk,ss_ticket_number,ss_sold_date_sk] InputAdapter BroadcastExchange #5 WholeStageCodegen (6) Filter [d_date_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.date_dim [d_date_sk] InputAdapter BroadcastExchange #6 WholeStageCodegen (7) Filter [s_store_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.store [s_store_sk,s_store_name,s_company_id,s_street_number,s_street_name,s_street_type,s_suite_number,s_city,s_county,s_state,s_zip] actual simplified plan: /home/runner/work/spark/spark/target/tmp/q50.sf100.actual.simplified.txt actual explain plan: /home/runner/work/spark/spark/target/tmp/q50.sf100.actual.explain.txt TakeOrderedAndProject [s_store_name,s_company_id,s_street_number,s_street_name,s_street_type,s_suite_number,s_city,s_county,s_state,s_zip,30 days ,31 - 60 days ,61 - 90 days ,91 - 120 days ,>120 days ] WholeStageCodegen (9) HashAggregate [s_store_name,s_company_id,s_street_number,s_street_name,s_street_type,s_suite_number,s_city,s_county,s_state,s_zip,sum,sum,sum,sum,sum] [sum(CASE WHEN ((sr_returned_date_sk - ss_sold_date_sk) <= 30) THEN 1 ELSE 0 END),sum(CASE WHEN (((sr_returned_date_sk - ss_sold_date_sk) > 30) AND ((sr_returned_date_sk - ss_sold_date_sk) <= 60)) THEN 1 ELSE 0 END),sum(CASE WHEN (((sr_returned_date_sk - ss_sold_date_sk) > 60) AND ((sr_returned_date_sk - ss_sold_date_sk) <= 90)) THEN 1 ELSE 0 END),sum(CASE WHEN (((sr_returned_date_sk - ss_sold_date_sk) > 90) AND ((sr_returned_date_sk - ss_sold_date_sk) <= 120)) THEN 1 ELSE 0 END),sum(CASE WHEN ((sr_returned_date_sk - ss_sold_date_sk) > 120) THEN 1 ELSE 0 END),30 days ,31 - 60 days ,61 - 90 days ,91 - 120 days ,>120 days ,sum,sum,sum,sum,sum] InputAdapter Exchange [s_store_name,s_company_id,s_street_number,s_street_name,s_street_type,s_suite_number,s_city,s_county,s_state,s_zip] #1 WholeStageCodegen (8) HashAggregate [s_store_name,s_company_id,s_street_number,s_street_name,s_street_type,s_suite_number,s_city,s_county,s_state,s_zip,sr_returned_date_sk,ss_sold_date_sk] [sum,sum,sum,sum,sum,sum,sum,sum,sum,sum] Project [ss_sold_date_sk,sr_returned_date_sk,s_store_name,s_company_id,s_street_number,s_street_name,s_street_type,s_suite_number,s_city,s_county,s_state,s_zip] BroadcastHashJoin [ss_store_sk,s_store_sk] Project [sr_returned_date_sk,ss_store_sk,ss_sold_date_sk] BroadcastHashJoin [ss_sold_date_sk,d_date_sk] Project [sr_returned_date_sk,ss_store_sk,ss_sold_date_sk] SortMergeJoin [sr_ticket_number,sr_item_sk,sr_customer_sk,ss_ticket_number,ss_item_sk,ss_customer_sk] InputAdapter WholeStageCodegen (3) Sort [sr_ticket_number,sr_item_sk,sr_customer_sk] InputAdapter Exchange [sr_ticket_number,sr_item_sk,sr_customer_sk] #2 WholeStageCodegen (2) Project [sr_item_sk,sr_customer_sk,sr_ticket_number,sr_returned_date_sk] BroadcastHashJoin [sr_returned_date_sk,d_date_sk] Filter [sr_ticket_number,sr_item_sk,sr_customer_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.store_returns [sr_item_sk,sr_customer_sk,sr_ticket_number,sr_returned_date_sk] SubqueryBroadcast [d_date_sk] #1 BroadcastExchange #3 WholeStageCodegen (1) Project [d_date_sk] Filter [d_year,d_moy,d_date_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.date_dim [d_date_sk,d_year,d_moy] InputAdapter ReusedExchange [d_date_sk] #3 InputAdapter WholeStageCodegen (5) Sort [ss_ticket_number,ss_item_sk,ss_customer_sk] InputAdapter Exchange [ss_ticket_number,ss_item_sk,ss_customer_sk] #4 WholeStageCodegen (4) Filter [ss_ticket_number,ss_item_sk,ss_customer_sk,ss_store_sk] Subquery #2 ObjectHashAggregate [buf] [bloom_filter_agg(xxhash64(sr_ticket_number, 42), 443806, 5270048, 0, 0),bloomFilter,buf] Exchange #5 ObjectHashAggregate [sr_ticket_number] [buf,buf] WholeStageCodegen (2) Project [sr_ticket_number] BroadcastHashJoin [sr_returned_date_sk,d_date_sk] Project [sr_ticket_number,sr_returned_date_sk] Filter [sr_ticket_number,sr_item_sk,sr_customer_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.store_returns [sr_item_sk,sr_customer_sk,sr_ticket_number,sr_returned_date_sk] ReusedSubquery [d_date_sk] #1 InputAdapter ReusedExchange [d_date_sk] #3 Subquery #3 ObjectHashAggregate [buf] [bloom_filter_agg(xxhash64(sr_item_sk, 42), 443806, 5270048, 0, 0),bloomFilter,buf] Exchange #6 ObjectHashAggregate [sr_item_sk] [buf,buf] WholeStageCodegen (2) Project [sr_item_sk] BroadcastHashJoin [sr_returned_date_sk,d_date_sk] Project [sr_item_sk,sr_returned_date_sk] Filter [sr_ticket_number,sr_item_sk,sr_customer_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.store_returns [sr_item_sk,sr_customer_sk,sr_ticket_number,sr_returned_date_sk] ReusedSubquery [d_date_sk] #1 InputAdapter ReusedExchange [d_date_sk] #3 Subquery #4 ObjectHashAggregate [buf] [bloom_filter_agg(xxhash64(sr_customer_sk, 42), 443806, 5270048, 0, 0),bloomFilter,buf] Exchange #7 ObjectHashAggregate [sr_customer_sk] [buf,buf] WholeStageCodegen (2) Project [sr_customer_sk] BroadcastHashJoin [sr_returned_date_sk,d_date_sk] Project [sr_customer_sk,sr_returned_date_sk] Filter [sr_ticket_number,sr_item_sk,sr_customer_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.store_returns [sr_item_sk,sr_customer_sk,sr_ticket_number,sr_returned_date_sk] ReusedSubquery [d_date_sk] #1 InputAdapter ReusedExchange [d_date_sk] #3 ColumnarToRow InputAdapter Scan parquet spark_catalog.default.store_sales [ss_item_sk,ss_customer_sk,ss_store_sk,ss_ticket_number,ss_sold_date_sk] InputAdapter BroadcastExchange #8 WholeStageCodegen (6) Filter [d_date_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.date_dim [d_date_sk] InputAdapter BroadcastExchange #9 WholeStageCodegen (7) Filter [s_store_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.store [s_store_sk,s_store_name,s_company_id,s_street_number,s_street_name,s_street_type,s_suite_number,s_city,s_county,s_state,s_zip]
TPCDSV1_4_PlanStabilityWithStatsSuite.check simplified sf100 (tpcds-v1.4/q54): TPCDSV1_4_PlanStabilityWithStatsSuite#L290
org.scalatest.exceptions.TestFailedException: Plans did not match: last approved simplified plan: /home/runner/work/spark/spark/sql/core/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q54.sf100/simplified.txt last approved explain plan: /home/runner/work/spark/spark/sql/core/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q54.sf100/explain.txt TakeOrderedAndProject [segment,num_customers,segment_base] WholeStageCodegen (16) HashAggregate [segment,count] [count(1),num_customers,segment_base,count] InputAdapter Exchange [segment] #1 WholeStageCodegen (15) HashAggregate [segment] [count,count] HashAggregate [c_customer_sk,sum] [sum(UnscaledValue(ss_ext_sales_price)),segment,sum] HashAggregate [c_customer_sk,ss_ext_sales_price] [sum,sum] Project [c_customer_sk,ss_ext_sales_price] SortMergeJoin [c_customer_sk,ss_customer_sk] InputAdapter WholeStageCodegen (11) Sort [c_customer_sk] Project [c_customer_sk] BroadcastHashJoin [ca_address_sk,c_current_addr_sk] InputAdapter BroadcastExchange #2 WholeStageCodegen (2) Project [ca_address_sk] BroadcastHashJoin [ca_county,ca_state,s_county,s_state] Filter [ca_address_sk,ca_county,ca_state] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.customer_address [ca_address_sk,ca_county,ca_state] InputAdapter BroadcastExchange #3 WholeStageCodegen (1) Filter [s_county,s_state] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.store [s_county,s_state] HashAggregate [c_customer_sk,c_current_addr_sk] HashAggregate [c_customer_sk,c_current_addr_sk] Project [c_customer_sk,c_current_addr_sk] SortMergeJoin [customer_sk,c_customer_sk] InputAdapter WholeStageCodegen (8) Sort [customer_sk] InputAdapter Exchange [customer_sk] #4 WholeStageCodegen (7) Project [customer_sk] BroadcastHashJoin [item_sk,i_item_sk] Project [customer_sk,item_sk] BroadcastHashJoin [sold_date_sk,d_date_sk] InputAdapter Union WholeStageCodegen (3) Project [cs_sold_date_sk,cs_bill_customer_sk,cs_item_sk] Filter [cs_item_sk,cs_bill_customer_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.catalog_sales [cs_bill_customer_sk,cs_item_sk,cs_sold_date_sk] SubqueryBroadcast [d_date_sk] #1 BroadcastExchange #5 WholeStageCodegen (1) Project [d_date_sk] Filter [d_moy,d_year,d_date_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.date_dim [d_date_sk,d_year,d_moy] WholeStageCodegen (4) Project [ws_sold_date_sk,ws_bill_customer_sk,ws_item_sk] Filter [ws_item_sk,ws_bill_customer_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.web_sales [ws_item_sk,ws_bill_customer_sk,ws_sold_date_sk] ReusedSubquery [d_date_sk] #1 InputAdapter ReusedExchange [d_date_sk] #5 InputAdapter BroadcastExchange #6 WholeStageCodegen (6) Project [i_item_sk] Filter [i_category,i_class,i_item_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.item [i_item_sk,i_class,i_category] InputAdapter WholeStageCodegen (10) Sort [c_customer_sk] InputAdapter Exchange [c_customer_sk] #7 WholeStageCodegen (9) Filter [c_customer_sk,c_current_addr_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.customer [c_customer_sk,c_current_addr_sk] InputAdapter WholeStageCodegen (14) Sort [ss_customer_sk] InputAdapter Exchange [ss_customer_sk] #8 WholeStageCodegen (13) Project [ss_customer_sk,ss_ext_sales_price] BroadcastHashJoin [ss_sold_date_sk,d_date_sk] Filter [ss_customer_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.store_sales [ss_customer_sk,ss_ext_sales_price,ss_sold_date_sk] SubqueryBroadcast [d_date_sk] #2 BroadcastExchange #9 WholeStageCodegen (1) Project [d_date_sk] Filter [d_month_seq,d_date_sk] ReusedSubquery [(d_month_seq + 1)] #3 ReusedSubquery [(d_month_seq + 3)] #4 ColumnarToRow InputAdapter Scan parquet spark_catalog.default.date_dim [d_date_sk,d_month_seq] Subquery #3 WholeStageCodegen (2) HashAggregate [(d_month_seq + 1)] InputAdapter Exchange [(d_month_seq + 1)] #10 WholeStageCodegen (1) HashAggregate [(d_month_seq + 1)] Project [d_month_seq] Filter [d_year,d_moy] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.date_dim [d_month_seq,d_year,d_moy] Subquery #4 WholeStageCodegen (2) HashAggregate [(d_month_seq + 3)] InputAdapter Exchange [(d_month_seq + 3)] #11 WholeStageCodegen (1) HashAggregate [(d_month_seq + 3)] Project [d_month_seq] Filter [d_year,d_moy] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.date_dim [d_month_seq,d_year,d_moy] InputAdapter ReusedExchange [d_date_sk] #9 actual simplified plan: /home/runner/work/spark/spark/target/tmp/q54.sf100.actual.simplified.txt actual explain plan: /home/runner/work/spark/spark/target/tmp/q54.sf100.actual.explain.txt TakeOrderedAndProject [segment,num_customers,segment_base] WholeStageCodegen (16) HashAggregate [segment,count] [count(1),num_customers,segment_base,count] InputAdapter Exchange [segment] #1 WholeStageCodegen (15) HashAggregate [segment] [count,count] HashAggregate [c_customer_sk,sum] [sum(UnscaledValue(ss_ext_sales_price)),segment,sum] HashAggregate [c_customer_sk,ss_ext_sales_price] [sum,sum] Project [c_customer_sk,ss_ext_sales_price] SortMergeJoin [c_customer_sk,ss_customer_sk] InputAdapter WholeStageCodegen (11) Sort [c_customer_sk] Project [c_customer_sk] BroadcastHashJoin [ca_address_sk,c_current_addr_sk] InputAdapter BroadcastExchange #2 WholeStageCodegen (2) Project [ca_address_sk] BroadcastHashJoin [ca_county,ca_state,s_county,s_state] Filter [ca_address_sk,ca_county,ca_state] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.customer_address [ca_address_sk,ca_county,ca_state] InputAdapter BroadcastExchange #3 WholeStageCodegen (1) Filter [s_county,s_state] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.store [s_county,s_state] HashAggregate [c_customer_sk,c_current_addr_sk] HashAggregate [c_customer_sk,c_current_addr_sk] Project [c_customer_sk,c_current_addr_sk] SortMergeJoin [customer_sk,c_customer_sk] InputAdapter WholeStageCodegen (8) Sort [customer_sk] InputAdapter Exchange [customer_sk] #4 WholeStageCodegen (7) Project [customer_sk] BroadcastHashJoin [item_sk,i_item_sk] Project [customer_sk,item_sk] BroadcastHashJoin [sold_date_sk,d_date_sk] InputAdapter Union WholeStageCodegen (3) Project [cs_sold_date_sk,cs_bill_customer_sk,cs_item_sk] Filter [cs_item_sk,cs_bill_customer_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.catalog_sales [cs_bill_customer_sk,cs_item_sk,cs_sold_date_sk] SubqueryBroadcast [d_date_sk] #1 BroadcastExchange #5 WholeStageCodegen (1) Project [d_date_sk] Filter [d_moy,d_year,d_date_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.date_dim [d_date_sk,d_year,d_moy] WholeStageCodegen (4) Project [ws_sold_date_sk,ws_bill_customer_sk,ws_item_sk] Filter [ws_item_sk,ws_bill_customer_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.web_sales [ws_item_sk,ws_bill_customer_sk,ws_sold_date_sk] ReusedSubquery [d_date_sk] #1 InputAdapter ReusedExchange [d_date_sk] #5 InputAdapter BroadcastExchange #6 WholeStageCodegen (6) Project [i_item_sk] Filter [i_category,i_class,i_item_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.item [i_item_sk,i_class,i_category] InputAdapter WholeStageCodegen (10) Sort [c_customer_sk] InputAdapter Exchange [c_customer_sk] #7 WholeStageCodegen (9) Filter [c_customer_sk,c_current_addr_sk] Subquery #2 ObjectHashAggregate [buf] [bloom_filter_agg(xxhash64(ss_customer_sk, 42), 262672542, 67108864, 0, 0),bloomFilter,buf] Exchange #8 ObjectHashAggregate [ss_customer_sk] [buf,buf] WholeStageCodegen (2) Project [ss_customer_sk] BroadcastHashJoin [ss_sold_date_sk,d_date_sk] Filter [ss_customer_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.store_sales [ss_customer_sk,ss_sold_date_sk] SubqueryBroadcast [d_date_sk] #3 BroadcastExchange #9 WholeStageCodegen (1) Project [d_date_sk] Filter [d_month_seq,d_date_sk] ReusedSubquery [(d_month_seq + 1)] #4 ReusedSubquery [(d_month_seq + 3)] #5 ColumnarToRow InputAdapter Scan parquet spark_catalog.default.date_dim [d_date_sk,d_month_seq] Subquery #4 WholeStageCodegen (2) HashAggregate [(d_month_seq + 1)] InputAdapter Exchange [(d_month_seq + 1)] #10 WholeStageCodegen (1) HashAggregate [(d_month_seq + 1)] Project [d_month_seq] Filter [d_year,d_moy] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.date_dim [d_month_seq,d_year,d_moy] Subquery #5 WholeStageCodegen (2) HashAggregate [(d_month_seq + 3)] InputAdapter Exchange [(d_month_seq + 3)] #11 WholeStageCodegen (1) HashAggregate [(d_month_seq + 3)] Project [d_month_seq] Filter [d_year,d_moy] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.date_dim [d_month_seq,d_year,d_moy] InputAdapter ReusedExchange [d_date_sk] #9 ColumnarToRow InputAdapter Scan parquet spark_catalog.default.customer [c_customer_sk,c_current_addr_sk] InputAdapter WholeStageCodegen (14) Sort [ss_customer_sk] InputAdapter Exchange [ss_customer_sk] #12 WholeStageCodegen (13) Project [ss_customer_sk,ss_ext_sales_price] BroadcastHashJoin [ss_sold_date_sk,d_date_sk] Filter [ss_customer_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.store_sales [ss_customer_sk,ss_ext_sales_price,ss_sold_date_sk] ReusedSubquery [d_date_sk] #3 InputAdapter ReusedExchange [d_date_sk] #9
TPCDSV1_4_PlanStabilityWithStatsSuite.check simplified sf100 (tpcds-v1.4/q70): TPCDSV1_4_PlanStabilityWithStatsSuite#L290
org.scalatest.exceptions.TestFailedException: Plans did not match: last approved simplified plan: /home/runner/work/spark/spark/sql/core/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q70.sf100/simplified.txt last approved explain plan: /home/runner/work/spark/spark/sql/core/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q70.sf100/explain.txt TakeOrderedAndProject [lochierarchy,s_state,rank_within_parent,total_sum,s_county] WholeStageCodegen (11) Project [total_sum,s_state,s_county,lochierarchy,rank_within_parent] InputAdapter Window [_w0,_w1,_w2] WholeStageCodegen (10) Sort [_w1,_w2,_w0] InputAdapter Exchange [_w1,_w2] #1 WholeStageCodegen (9) HashAggregate [s_state,s_county,spark_grouping_id,sum] [sum(UnscaledValue(ss_net_profit)),total_sum,lochierarchy,_w0,_w1,_w2,sum] InputAdapter Exchange [s_state,s_county,spark_grouping_id] #2 WholeStageCodegen (8) HashAggregate [s_state,s_county,spark_grouping_id,ss_net_profit] [sum,sum] Expand [ss_net_profit,s_state,s_county] Project [ss_net_profit,s_state,s_county] BroadcastHashJoin [ss_store_sk,s_store_sk] Project [ss_store_sk,ss_net_profit] BroadcastHashJoin [ss_sold_date_sk,d_date_sk] Filter [ss_store_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.store_sales [ss_store_sk,ss_net_profit,ss_sold_date_sk] SubqueryBroadcast [d_date_sk] #1 BroadcastExchange #3 WholeStageCodegen (1) Project [d_date_sk] Filter [d_month_seq,d_date_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.date_dim [d_date_sk,d_month_seq] InputAdapter ReusedExchange [d_date_sk] #3 InputAdapter BroadcastExchange #4 WholeStageCodegen (7) BroadcastHashJoin [s_state,s_state] Filter [s_store_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.store [s_store_sk,s_county,s_state] InputAdapter BroadcastExchange #5 WholeStageCodegen (6) Project [s_state] Filter [ranking] InputAdapter Window [_w0,s_state] WindowGroupLimit [s_state,_w0] WholeStageCodegen (5) Sort [s_state,_w0] HashAggregate [sum] [sum(UnscaledValue(ss_net_profit)),_w0,s_state,sum] InputAdapter Exchange [s_state] #6 WholeStageCodegen (4) HashAggregate [s_state,ss_net_profit] [sum,sum] Project [ss_net_profit,s_state] BroadcastHashJoin [ss_store_sk,s_store_sk] Project [ss_store_sk,ss_net_profit] BroadcastHashJoin [ss_sold_date_sk,d_date_sk] Filter [ss_store_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.store_sales [ss_store_sk,ss_net_profit,ss_sold_date_sk] ReusedSubquery [d_date_sk] #1 InputAdapter ReusedExchange [d_date_sk] #3 InputAdapter BroadcastExchange #7 WholeStageCodegen (3) Filter [s_store_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.store [s_store_sk,s_state] actual simplified plan: /home/runner/work/spark/spark/target/tmp/q70.sf100.actual.simplified.txt actual explain plan: /home/runner/work/spark/spark/target/tmp/q70.sf100.actual.explain.txt TakeOrderedAndProject [lochierarchy,s_state,rank_within_parent,total_sum,s_county] WholeStageCodegen (11) Project [total_sum,s_state,s_county,lochierarchy,rank_within_parent] InputAdapter Window [_w0,_w1,_w2] WholeStageCodegen (10) Sort [_w1,_w2,_w0] InputAdapter Exchange [_w1,_w2] #1 WholeStageCodegen (9) HashAggregate [s_state,s_county,spark_grouping_id,sum] [sum(UnscaledValue(ss_net_profit)),total_sum,lochierarchy,_w0,_w1,_w2,sum] InputAdapter Exchange [s_state,s_county,spark_grouping_id] #2 WholeStageCodegen (8) HashAggregate [s_state,s_county,spark_grouping_id,ss_net_profit] [sum,sum] Expand [ss_net_profit,s_state,s_county] Project [ss_net_profit,s_state,s_county] BroadcastHashJoin [ss_store_sk,s_store_sk] Project [ss_store_sk,ss_net_profit] BroadcastHashJoin [ss_sold_date_sk,d_date_sk] Filter [ss_store_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.store_sales [ss_store_sk,ss_net_profit,ss_sold_date_sk] SubqueryBroadcast [d_date_sk] #1 BroadcastExchange #3 WholeStageCodegen (1) Project [d_date_sk] Filter [d_month_seq,d_date_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.date_dim [d_date_sk,d_month_seq] InputAdapter ReusedExchange [d_date_sk] #3 InputAdapter BroadcastExchange #4 WholeStageCodegen (7) BroadcastHashJoin [s_state,s_state] Filter [s_store_sk] Subquery #2 ObjectHashAggregate [buf] [bloom_filter_agg(xxhash64(ss_store_sk, 42), 54151237, 67108864, 0, 0),bloomFilter,buf] Exchange #5 ObjectHashAggregate [ss_store_sk] [buf,buf] WholeStageCodegen (2) Project [ss_store_sk] BroadcastHashJoin [ss_sold_date_sk,d_date_sk] Filter [ss_store_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.store_sales [ss_store_sk,ss_sold_date_sk] ReusedSubquery [d_date_sk] #1 InputAdapter ReusedExchange [d_date_sk] #3 ColumnarToRow InputAdapter Scan parquet spark_catalog.default.store [s_store_sk,s_county,s_state] InputAdapter BroadcastExchange #6 WholeStageCodegen (6) Project [s_state] Filter [ranking] InputAdapter Window [_w0,s_state] WindowGroupLimit [s_state,_w0] WholeStageCodegen (5) Sort [s_state,_w0] HashAggregate [sum] [sum(UnscaledValue(ss_net_profit)),_w0,s_state,sum] InputAdapter Exchange [s_state] #7 WholeStageCodegen (4) HashAggregate [s_state,ss_net_profit] [sum,sum] Project [ss_net_profit,s_state] BroadcastHashJoin [ss_store_sk,s_store_sk] Project [ss_store_sk,ss_net_profit] BroadcastHashJoin [ss_sold_date_sk,d_date_sk] Filter [ss_store_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.store_sales [ss_store_sk,ss_net_profit,ss_sold_date_sk] ReusedSubquery [d_date_sk] #1 InputAdapter ReusedExchange [d_date_sk] #3 InputAdapter BroadcastExchange #8 WholeStageCodegen (3) Filter [s_store_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.store [s_store_sk,s_state]
TPCDSV1_4_PlanStabilityWithStatsSuite.check simplified sf100 (tpcds-v1.4/q93): TPCDSV1_4_PlanStabilityWithStatsSuite#L290
org.scalatest.exceptions.TestFailedException: Plans did not match: last approved simplified plan: /home/runner/work/spark/spark/sql/core/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q93.sf100/simplified.txt last approved explain plan: /home/runner/work/spark/spark/sql/core/src/test/resources/tpcds-plan-stability/approved-plans-v1_4/q93.sf100/explain.txt TakeOrderedAndProject [sumsales,ss_customer_sk] WholeStageCodegen (7) HashAggregate [ss_customer_sk,sum,isEmpty] [sum(act_sales),sumsales,sum,isEmpty] InputAdapter Exchange [ss_customer_sk] #1 WholeStageCodegen (6) HashAggregate [ss_customer_sk,act_sales] [sum,isEmpty,sum,isEmpty] Project [ss_customer_sk,sr_return_quantity,ss_quantity,ss_sales_price] SortMergeJoin [sr_item_sk,sr_ticket_number,ss_item_sk,ss_ticket_number] InputAdapter WholeStageCodegen (3) Sort [sr_item_sk,sr_ticket_number] InputAdapter Exchange [sr_item_sk,sr_ticket_number] #2 WholeStageCodegen (2) Project [sr_item_sk,sr_ticket_number,sr_return_quantity] BroadcastHashJoin [sr_reason_sk,r_reason_sk] Project [sr_item_sk,sr_reason_sk,sr_ticket_number,sr_return_quantity] Filter [sr_item_sk,sr_ticket_number,sr_reason_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.store_returns [sr_item_sk,sr_reason_sk,sr_ticket_number,sr_return_quantity,sr_returned_date_sk] InputAdapter BroadcastExchange #3 WholeStageCodegen (1) Project [r_reason_sk] Filter [r_reason_desc,r_reason_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.reason [r_reason_sk,r_reason_desc] InputAdapter WholeStageCodegen (5) Sort [ss_item_sk,ss_ticket_number] InputAdapter Exchange [ss_item_sk,ss_ticket_number] #4 WholeStageCodegen (4) Project [ss_item_sk,ss_customer_sk,ss_ticket_number,ss_quantity,ss_sales_price] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.store_sales [ss_item_sk,ss_customer_sk,ss_ticket_number,ss_quantity,ss_sales_price,ss_sold_date_sk] actual simplified plan: /home/runner/work/spark/spark/target/tmp/q93.sf100.actual.simplified.txt actual explain plan: /home/runner/work/spark/spark/target/tmp/q93.sf100.actual.explain.txt TakeOrderedAndProject [sumsales,ss_customer_sk] WholeStageCodegen (7) HashAggregate [ss_customer_sk,sum,isEmpty] [sum(act_sales),sumsales,sum,isEmpty] InputAdapter Exchange [ss_customer_sk] #1 WholeStageCodegen (6) HashAggregate [ss_customer_sk,act_sales] [sum,isEmpty,sum,isEmpty] Project [ss_customer_sk,sr_return_quantity,ss_quantity,ss_sales_price] SortMergeJoin [sr_item_sk,sr_ticket_number,ss_item_sk,ss_ticket_number] InputAdapter WholeStageCodegen (3) Sort [sr_item_sk,sr_ticket_number] InputAdapter Exchange [sr_item_sk,sr_ticket_number] #2 WholeStageCodegen (2) Project [sr_item_sk,sr_ticket_number,sr_return_quantity] BroadcastHashJoin [sr_reason_sk,r_reason_sk] Project [sr_item_sk,sr_reason_sk,sr_ticket_number,sr_return_quantity] Filter [sr_item_sk,sr_ticket_number,sr_reason_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.store_returns [sr_item_sk,sr_reason_sk,sr_ticket_number,sr_return_quantity,sr_returned_date_sk] InputAdapter BroadcastExchange #3 WholeStageCodegen (1) Project [r_reason_sk] Filter [r_reason_desc,r_reason_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.reason [r_reason_sk,r_reason_desc] InputAdapter WholeStageCodegen (5) Sort [ss_item_sk,ss_ticket_number] InputAdapter Exchange [ss_item_sk,ss_ticket_number] #4 WholeStageCodegen (4) Project [ss_item_sk,ss_customer_sk,ss_ticket_number,ss_quantity,ss_sales_price] Filter [ss_item_sk,ss_ticket_number] Subquery #1 ObjectHashAggregate [buf] [bloom_filter_agg(xxhash64(sr_item_sk, 42), 496193, 5776894, 0, 0),bloomFilter,buf] Exchange #5 ObjectHashAggregate [sr_item_sk] [buf,buf] WholeStageCodegen (2) Project [sr_item_sk] BroadcastHashJoin [sr_reason_sk,r_reason_sk] Project [sr_item_sk,sr_reason_sk] Filter [sr_item_sk,sr_ticket_number,sr_reason_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.store_returns [sr_item_sk,sr_reason_sk,sr_ticket_number,sr_returned_date_sk] InputAdapter ReusedExchange [r_reason_sk] #3 Subquery #2 ObjectHashAggregate [buf] [bloom_filter_agg(xxhash64(sr_ticket_number, 42), 496193, 5776894, 0, 0),bloomFilter,buf] Exchange #6 ObjectHashAggregate [sr_ticket_number] [buf,buf] WholeStageCodegen (2) Project [sr_ticket_number] BroadcastHashJoin [sr_reason_sk,r_reason_sk] Project [sr_reason_sk,sr_ticket_number] Filter [sr_item_sk,sr_ticket_number,sr_reason_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.store_returns [sr_item_sk,sr_reason_sk,sr_ticket_number,sr_returned_date_sk] InputAdapter ReusedExchange [r_reason_sk] #3 ColumnarToRow InputAdapter Scan parquet spark_catalog.default.store_sales [ss_item_sk,ss_customer_sk,ss_ticket_number,ss_quantity,ss_sales_price,ss_sold_date_sk]
TPCDSV2_7_PlanStabilityWithStatsSuite.check simplified sf100 (tpcds-v2.7.0/q14): TPCDSV2_7_PlanStabilityWithStatsSuite#L316
org.scalatest.exceptions.TestFailedException: Plans did not match: last approved simplified plan: /home/runner/work/spark/spark/sql/core/src/test/resources/tpcds-plan-stability/approved-plans-v2_7/q14.sf100/simplified.txt last approved explain plan: /home/runner/work/spark/spark/sql/core/src/test/resources/tpcds-plan-stability/approved-plans-v2_7/q14.sf100/explain.txt TakeOrderedAndProject [i_brand_id,i_class_id,i_category_id,channel,sales,number_sales,channel,i_brand_id,i_class_id,i_category_id,sales,number_sales] WholeStageCodegen (76) BroadcastHashJoin [i_brand_id,i_class_id,i_category_id,i_brand_id,i_class_id,i_category_id] Filter [sales] Subquery #4 WholeStageCodegen (8) HashAggregate [sum,count] [avg((cast(quantity as decimal(10,0)) * list_price)),average_sales,sum,count] InputAdapter Exchange #14 WholeStageCodegen (7) HashAggregate [quantity,list_price] [sum,count,sum,count] InputAdapter Union WholeStageCodegen (2) Project [ss_quantity,ss_list_price] BroadcastHashJoin [ss_sold_date_sk,d_date_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.store_sales [ss_quantity,ss_list_price,ss_sold_date_sk] ReusedSubquery [d_date_sk] #3 InputAdapter ReusedExchange [d_date_sk] #7 WholeStageCodegen (4) Project [cs_quantity,cs_list_price] BroadcastHashJoin [cs_sold_date_sk,d_date_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.catalog_sales [cs_quantity,cs_list_price,cs_sold_date_sk] ReusedSubquery [d_date_sk] #3 InputAdapter ReusedExchange [d_date_sk] #7 WholeStageCodegen (6) Project [ws_quantity,ws_list_price] BroadcastHashJoin [ws_sold_date_sk,d_date_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.web_sales [ws_quantity,ws_list_price,ws_sold_date_sk] ReusedSubquery [d_date_sk] #3 InputAdapter ReusedExchange [d_date_sk] #7 HashAggregate [i_brand_id,i_class_id,i_category_id,sum,isEmpty,count] [sum((cast(ss_quantity as decimal(10,0)) * ss_list_price)),count(1),channel,sales,number_sales,sum,isEmpty,count] InputAdapter Exchange [i_brand_id,i_class_id,i_category_id] #1 WholeStageCodegen (37) HashAggregate [i_brand_id,i_class_id,i_category_id,ss_quantity,ss_list_price] [sum,isEmpty,count,sum,isEmpty,count] Project [ss_quantity,ss_list_price,i_brand_id,i_class_id,i_category_id] BroadcastHashJoin [ss_item_sk,i_item_sk] Project [ss_item_sk,ss_quantity,ss_list_price] BroadcastHashJoin [ss_sold_date_sk,d_date_sk] BroadcastHashJoin [ss_item_sk,ss_item_sk] Filter [ss_item_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.store_sales [ss_item_sk,ss_quantity,ss_list_price,ss_sold_date_sk] SubqueryBroadcast [d_date_sk] #1 BroadcastExchange #2 WholeStageCodegen (1) Project [d_date_sk] Filter [d_week_seq,d_date_sk] ReusedSubquery [d_week_seq] #2 ColumnarToRow InputAdapter Scan parquet spark_catalog.default.date_dim [d_date_sk,d_week_seq] Subquery #2 WholeStageCodegen (1) Project [d_week_seq] Filter [d_year,d_moy,d_dom] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.date_dim [d_week_seq,d_year,d_moy,d_dom] InputAdapter BroadcastExchange #3 WholeStageCodegen (17) Project [i_item_sk] BroadcastHashJoin [i_brand_id,i_class_id,i_category_id,brand_id,class_id,category_id] Filter [i_brand_id,i_class_id,i_category_id] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.item [i_item_sk,i_brand_id,i_class_id,i_category_id] InputAdapter BroadcastExchange #4 WholeStageCodegen (16) SortMergeJoin [brand_id,class_id,category_id,i_brand_id,i_class_id,i_category_id] InputAdapter WholeStageCodegen (11) Sort [brand_id,class_id,category_id] InputAdapter Exchange [brand_id,class_id,category_id] #5 WholeStageCodegen (10) HashAggregate [brand_id,class_id,category_id] InputAdapter Exchange [brand_id,class_id,category_id] #6 WholeStageCodegen (9) HashAggregate [brand_id,class_id,category_id] Project [i_brand_id,i_class_id,i_category_id] BroadcastHashJoin [ss_item_sk,i_item_sk] Project [ss_item_sk] BroadcastHashJoin [ss_sold_date_sk,d_date_sk] Filter [ss_item_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.store_sales [ss_item_sk,ss_sold_date_sk] SubqueryBroadcast [d_date_sk] #3 BroadcastExchange #7 WholeStageCodegen (1) Project [d_date_sk] Filter [d_year,d_date_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.date_dim [d_date_sk,d_year] InputAdapter ReusedExchange [d_date_sk] #7 InputAdapter BroadcastExchange #8 WholeStageCodegen (8) SortMergeJoin [i_brand_id,i_class_id,i_category_id,i_brand_id,i_class_id,i_category_id] InputAdapter WholeStageCodegen (3) Sort [i_brand_id,i_class_id,i_category_id] InputAdapter Exchange [i_brand_id,i_class_id,i_category_id] #9 WholeStageCodegen (2) Filter [i_item_sk,i_brand_id,i_class_id,i_category_id] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.item [i_item_sk,i_brand_id,i_class_id,i_category_id] InputAdapter WholeStageCodegen (7) Sort [i_brand_id,i_class_id,i_category_id] InputAdapter Exchange [i_brand_id,i_class_id,i_category_id] #10 WholeStageCodegen (6) Project [i_brand_id,i_class_id,i_category_id] BroadcastHashJoin [cs_item_sk,i_item_sk] Project [cs_item_sk] BroadcastHashJoin [cs_sold_date_sk,d_date_sk] Filter [cs_item_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.catalog_sales [cs_item_sk,cs_sold_date_sk] ReusedSubquery [d_date_sk] #3 InputAdapter ReusedExchange [d_date_sk] #7 InputAdapter BroadcastExchange #11 WholeStageCodegen (5) Filter [i_item_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.item [i_item_sk,i_brand_id,i_class_id,i_category_id] InputAdapter WholeStageCodegen (15) Sort [i_brand_id,i_class_id,i_category_id] InputAdapter Exchange [i_brand_id,i_class_id,i_category_id] #12 WholeStageCodegen (14) Project [i_brand_id,i_class_id,i_category_id] BroadcastHashJoin [ws_item_sk,i_item_sk] Project [ws_item_sk] BroadcastHashJoin [ws_sold_date_sk,d_date_sk] Filter [ws_item_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.web_sales [ws_item_sk,ws_sold_date_sk] ReusedSubquery [d_date_sk] #3 InputAdapter ReusedExchange [d_date_sk] #7 InputAdapter ReusedExchange [i_item_sk,i_brand_id,i_class_id,i_category_id] #11 InputAdapter ReusedExchange [d_date_sk] #2 InputAdapter BroadcastExchange #13 WholeStageCodegen (36) BroadcastHashJoin [i_item_sk,ss_item_sk] Filter [i_item_sk,i_brand_id,i_class_id,i_category_id] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.item [i_item_sk,i_brand_id,i_class_id,i_category_id] InputAdapter ReusedExchange [ss_item_sk] #3 InputAdapter BroadcastExchange #15 WholeStageCodegen (75) Filter [sales] ReusedSubquery [average_sales] #4 HashAggregate [i_brand_id,i_class_id,i_category_id,sum,isEmpty,count] [sum((cast(ss_quantity as decimal(10,0)) * ss_list_price)),count(1),channel,sales,number_sales,sum,isEmpty,count] InputAdapter Exchange [i_brand_id,i_class_id,i_category_id] #16 WholeStageCodegen (74) HashAggregate [i_brand_id,i_class_id,i_category_id,ss_quantity,ss_list_price] [sum,isEmpty,count,sum,isEmpty,count] Project [ss_quantity,ss_list_price,i_brand_id,i_class_id,i_category_id] BroadcastHashJoin [ss_item_sk,i_item_sk] Project [ss_item_sk,ss_quantity,ss_list_price] BroadcastHashJoin [ss_sold_date_sk,d_date_sk] BroadcastHashJoin [ss_item_sk,ss_item_sk] Filter [ss_item_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.store_sales [ss_item_sk,ss_quantity,ss_list_price,ss_sold_date_sk] SubqueryBroadcast [d_date_sk] #5 BroadcastExchange #17 WholeStageCodegen (1) Project [d_date_sk] Filter [d_week_seq,d_date_sk] ReusedSubquery [d_week_seq] #6 ColumnarToRow InputAdapter Scan parquet spark_catalog.default.date_dim [d_date_sk,d_week_seq] Subquery #6 WholeStageCodegen (1) Project [d_week_seq] Filter [d_year,d_moy,d_dom] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.date_dim [d_week_seq,d_year,d_moy,d_dom] InputAdapter ReusedExchange [ss_item_sk] #3 InputAdapter ReusedExchange [d_date_sk] #17 InputAdapter ReusedExchange [i_item_sk,i_brand_id,i_class_id,i_category_id] #13 actual simplified plan: /home/runner/work/spark/spark/target/tmp/q14.sf100.actual.simplified.txt actual explain plan: /home/runner/work/spark/spark/target/tmp/q14.sf100.actual.explain.txt TakeOrderedAndProject [i_brand_id,i_class_id,i_category_id,channel,sales,number_sales,channel,i_brand_id,i_class_id,i_category_id,sales,number_sales] WholeStageCodegen (76) BroadcastHashJoin [i_brand_id,i_class_id,i_category_id,i_brand_id,i_class_id,i_category_id] Filter [sales] Subquery #5 WholeStageCodegen (8) HashAggregate [sum,count] [avg((cast(quantity as decimal(10,0)) * list_price)),average_sales,sum,count] InputAdapter Exchange #15 WholeStageCodegen (7) HashAggregate [quantity,list_price] [sum,count,sum,count] InputAdapter Union WholeStageCodegen (2) Project [ss_quantity,ss_list_price] BroadcastHashJoin [ss_sold_date_sk,d_date_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.store_sales [ss_quantity,ss_list_price,ss_sold_date_sk] ReusedSubquery [d_date_sk] #3 InputAdapter ReusedExchange [d_date_sk] #7 WholeStageCodegen (4) Project [cs_quantity,cs_list_price] BroadcastHashJoin [cs_sold_date_sk,d_date_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.catalog_sales [cs_quantity,cs_list_price,cs_sold_date_sk] ReusedSubquery [d_date_sk] #3 InputAdapter ReusedExchange [d_date_sk] #7 WholeStageCodegen (6) Project [ws_quantity,ws_list_price] BroadcastHashJoin [ws_sold_date_sk,d_date_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.web_sales [ws_quantity,ws_list_price,ws_sold_date_sk] ReusedSubquery [d_date_sk] #3 InputAdapter ReusedExchange [d_date_sk] #7 HashAggregate [i_brand_id,i_class_id,i_category_id,sum,isEmpty,count] [sum((cast(ss_quantity as decimal(10,0)) * ss_list_price)),count(1),channel,sales,number_sales,sum,isEmpty,count] InputAdapter Exchange [i_brand_id,i_class_id,i_category_id] #1 WholeStageCodegen (37) HashAggregate [i_brand_id,i_class_id,i_category_id,ss_quantity,ss_list_price] [sum,isEmpty,count,sum,isEmpty,count] Project [ss_quantity,ss_list_price,i_brand_id,i_class_id,i_category_id] BroadcastHashJoin [ss_item_sk,i_item_sk] Project [ss_item_sk,ss_quantity,ss_list_price] BroadcastHashJoin [ss_sold_date_sk,d_date_sk] BroadcastHashJoin [ss_item_sk,ss_item_sk] Filter [ss_item_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.store_sales [ss_item_sk,ss_quantity,ss_list_price,ss_sold_date_sk] SubqueryBroadcast [d_date_sk] #1 BroadcastExchange #2 WholeStageCodegen (1) Project [d_date_sk] Filter [d_week_seq,d_date_sk] ReusedSubquery [d_week_seq] #2 ColumnarToRow InputAdapter Scan parquet spark_catalog.default.date_dim [d_date_sk,d_week_seq] Subquery #2 WholeStageCodegen (1) Project [d_week_seq] Filter [d_year,d_moy,d_dom] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.date_dim [d_week_seq,d_year,d_moy,d_dom] InputAdapter BroadcastExchange #3 WholeStageCodegen (17) Project [i_item_sk] BroadcastHashJoin [i_brand_id,i_class_id,i_category_id,brand_id,class_id,category_id] Filter [i_brand_id,i_class_id,i_category_id] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.item [i_item_sk,i_brand_id,i_class_id,i_category_id] InputAdapter BroadcastExchange #4 WholeStageCodegen (16) SortMergeJoin [brand_id,class_id,category_id,i_brand_id,i_class_id,i_category_id] InputAdapter WholeStageCodegen (11) Sort [brand_id,class_id,category_id] InputAdapter Exchange [brand_id,class_id,category_id] #5 WholeStageCodegen (10) HashAggregate [brand_id,class_id,category_id] InputAdapter Exchange [brand_id,class_id,category_id] #6 WholeStageCodegen (9) HashAggregate [brand_id,class_id,category_id] Project [i_brand_id,i_class_id,i_category_id] BroadcastHashJoin [ss_item_sk,i_item_sk] Project [ss_item_sk] BroadcastHashJoin [ss_sold_date_sk,d_date_sk] Filter [ss_item_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.store_sales [ss_item_sk,ss_sold_date_sk] SubqueryBroadcast [d_date_sk] #3 BroadcastExchange #7 WholeStageCodegen (1) Project [d_date_sk] Filter [d_year,d_date_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.date_dim [d_date_sk,d_year] InputAdapter ReusedExchange [d_date_sk] #7 InputAdapter BroadcastExchange #8 WholeStageCodegen (8) SortMergeJoin [i_brand_id,i_class_id,i_category_id,i_brand_id,i_class_id,i_category_id] InputAdapter WholeStageCodegen (3) Sort [i_brand_id,i_class_id,i_category_id] InputAdapter Exchange [i_brand_id,i_class_id,i_category_id] #9 WholeStageCodegen (2) Filter [i_item_sk,i_brand_id,i_class_id,i_category_id] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.item [i_item_sk,i_brand_id,i_class_id,i_category_id] InputAdapter WholeStageCodegen (7) Sort [i_brand_id,i_class_id,i_category_id] InputAdapter Exchange [i_brand_id,i_class_id,i_category_id] #10 WholeStageCodegen (6) Project [i_brand_id,i_class_id,i_category_id] BroadcastHashJoin [cs_item_sk,i_item_sk] Project [cs_item_sk] BroadcastHashJoin [cs_sold_date_sk,d_date_sk] Filter [cs_item_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.catalog_sales [cs_item_sk,cs_sold_date_sk] ReusedSubquery [d_date_sk] #3 InputAdapter ReusedExchange [d_date_sk] #7 InputAdapter BroadcastExchange #11 WholeStageCodegen (5) Filter [i_item_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.item [i_item_sk,i_brand_id,i_class_id,i_category_id] InputAdapter WholeStageCodegen (15) Sort [i_brand_id,i_class_id,i_category_id] InputAdapter Exchange [i_brand_id,i_class_id,i_category_id] #12 WholeStageCodegen (14) Project [i_brand_id,i_class_id,i_category_id] BroadcastHashJoin [ws_item_sk,i_item_sk] Project [ws_item_sk] BroadcastHashJoin [ws_sold_date_sk,d_date_sk] Filter [ws_item_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.web_sales [ws_item_sk,ws_sold_date_sk] ReusedSubquery [d_date_sk] #3 InputAdapter ReusedExchange [d_date_sk] #7 InputAdapter ReusedExchange [i_item_sk,i_brand_id,i_class_id,i_category_id] #11 InputAdapter ReusedExchange [d_date_sk] #2 InputAdapter BroadcastExchange #13 WholeStageCodegen (36) BroadcastHashJoin [i_item_sk,ss_item_sk] Filter [i_item_sk,i_brand_id,i_class_id,i_category_id] Subquery #4 ObjectHashAggregate [buf] [bloom_filter_agg(xxhash64(ss_item_sk, 42), 275041999, 67108864, 0, 0),bloomFilter,buf] Exchange #14 ObjectHashAggregate [ss_item_sk] [buf,buf] WholeStageCodegen (19) Project [ss_item_sk] BroadcastHashJoin [ss_sold_date_sk,d_date_sk] Project [ss_item_sk,ss_sold_date_sk] Filter [ss_sold_date_sk] ReusedSubquery [d_date_sk] #1 BroadcastHashJoin [ss_item_sk,ss_item_sk] Filter [ss_item_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.store_sales [ss_item_sk,ss_sold_date_sk] InputAdapter ReusedExchange [ss_item_sk] #3 InputAdapter ReusedExchange [d_date_sk] #2 ColumnarToRow InputAdapter Scan parquet spark_catalog.default.item [i_item_sk,i_brand_id,i_class_id,i_category_id] InputAdapter ReusedExchange [ss_item_sk] #3 InputAdapter BroadcastExchange #16 WholeStageCodegen (75) Filter [sales] ReusedSubquery [average_sales] #5 HashAggregate [i_brand_id,i_class_id,i_category_id,sum,isEmpty,count] [sum((cast(ss_quantity as decimal(10,0)) * ss_list_price)),count(1),channel,sales,number_sales,sum,isEmpty,count] InputAdapter Exchange [i_brand_id,i_class_id,i_category_id] #17 WholeStageCodegen (74) HashAggregate [i_brand_id,i_class_id,i_category_id,ss_quantity,ss_list_price] [sum,isEmpty,count,sum,isEmpty,count] Project [ss_quantity,ss_list_price,i_brand_id,i_class_id,i_category_id] BroadcastHashJoin [ss_item_sk,i_item_sk] Project [ss_item_sk,ss_quantity,ss_list_price] BroadcastHashJoin [ss_sold_date_sk,d_date_sk] BroadcastHashJoin [ss_item_sk,ss_item_sk] Filter [ss_item_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.store_sales [ss_item_sk,ss_quantity,ss_list_price,ss_sold_date_sk] SubqueryBroadcast [d_date_sk] #6 BroadcastExchange #18 WholeStageCodegen (1) Project [d_date_sk] Filter [d_week_seq,d_date_sk] ReusedSubquery [d_week_seq] #7 ColumnarToRow InputAdapter Scan parquet spark_catalog.default.date_dim [d_date_sk,d_week_seq] Subquery #7 WholeStageCodegen (1) Project [d_week_seq] Filter [d_year,d_moy,d_dom] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.date_dim [d_week_seq,d_year,d_moy,d_dom] InputAdapter ReusedExchange [ss_item_sk] #3 InputAdapter ReusedExchange [d_date_sk] #18 InputAdapter BroadcastExchange #19 WholeStageCodegen (73) BroadcastHashJoin [i_item_sk,ss_item_sk] Filter [i_item_sk,i_brand_id,i_class_id,i_category_id] Subquery #8 ObjectHashAggregate [buf] [bloom_filter_agg(xxhash64(ss_item_sk, 42), 275041999, 67108864, 0, 0),bloomFilter,buf] Exchange #20 ObjectHashAggregate [ss_item_sk] [buf,buf] WholeStageCodegen (19) Project [ss_item_sk] BroadcastHashJoin [ss_sold_date_sk,d_date_sk] Project [ss_item_sk,ss_sold_date_sk] Filter [ss_sold_date_sk] ReusedSubquery [d_date_sk] #6 BroadcastHashJoin [ss_item_sk,ss_item_sk] Filter [ss_item_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.store_sales [ss_item_sk,ss_sold_date_sk] InputAdapter ReusedExchange [ss_item_sk] #3 InputAdapter ReusedExchange [d_date_sk] #18 ColumnarToRow InputAdapter Scan parquet spark_catalog.default.item [i_item_sk,i_brand_id,i_class_id,i_category_id] InputAdapter ReusedExchange [ss_item_sk] #3
TPCDSV2_7_PlanStabilityWithStatsSuite.check simplified sf100 (tpcds-v2.7.0/q14a): TPCDSV2_7_PlanStabilityWithStatsSuite#L316
org.scalatest.exceptions.TestFailedException: Plans did not match: last approved simplified plan: /home/runner/work/spark/spark/sql/core/src/test/resources/tpcds-plan-stability/approved-plans-v2_7/q14a.sf100/simplified.txt last approved explain plan: /home/runner/work/spark/spark/sql/core/src/test/resources/tpcds-plan-stability/approved-plans-v2_7/q14a.sf100/explain.txt TakeOrderedAndProject [channel,i_brand_id,i_class_id,i_category_id,sum_sales,number_sales] WholeStageCodegen (586) HashAggregate [channel,i_brand_id,i_class_id,i_category_id,sum_sales,number_sales] InputAdapter Exchange [channel,i_brand_id,i_class_id,i_category_id,sum_sales,number_sales] #1 WholeStageCodegen (585) HashAggregate [channel,i_brand_id,i_class_id,i_category_id,sum_sales,number_sales] InputAdapter Union WholeStageCodegen (116) HashAggregate [channel,i_brand_id,i_class_id,i_category_id,sum,isEmpty,sum] [sum(sales),sum(number_salesL),sum_sales,number_sales,sum,isEmpty,sum] InputAdapter Exchange [channel,i_brand_id,i_class_id,i_category_id] #2 WholeStageCodegen (115) HashAggregate [channel,i_brand_id,i_class_id,i_category_id,sales,number_sales] [sum,isEmpty,sum,sum,isEmpty,sum] InputAdapter Union WholeStageCodegen (38) Filter [sales] Subquery #3 WholeStageCodegen (8) HashAggregate [sum,count] [avg((cast(quantity as decimal(10,0)) * list_price)),average_sales,sum,count] InputAdapter Exchange #16 WholeStageCodegen (7) HashAggregate [quantity,list_price] [sum,count,sum,count] InputAdapter Union WholeStageCodegen (2) Project [ss_quantity,ss_list_price] BroadcastHashJoin [ss_sold_date_sk,d_date_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.store_sales [ss_quantity,ss_list_price,ss_sold_date_sk] ReusedSubquery [d_date_sk] #2 InputAdapter ReusedExchange [d_date_sk] #9 WholeStageCodegen (4) Project [cs_quantity,cs_list_price] BroadcastHashJoin [cs_sold_date_sk,d_date_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.catalog_sales [cs_quantity,cs_list_price,cs_sold_date_sk] SubqueryBroadcast [d_date_sk] #4 BroadcastExchange #17 WholeStageCodegen (1) Project [d_date_sk] Filter [d_year,d_date_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.date_dim [d_date_sk,d_year] InputAdapter ReusedExchange [d_date_sk] #17 WholeStageCodegen (6) Project [ws_quantity,ws_list_price] BroadcastHashJoin [ws_sold_date_sk,d_date_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.web_sales [ws_quantity,ws_list_price,ws_sold_date_sk] ReusedSubquery [d_date_sk] #4 InputAdapter ReusedExchange [d_date_sk] #17 HashAggregate [i_brand_id,i_class_id,i_category_id,sum,isEmpty,count] [sum((cast(ss_quantity as decimal(10,0)) * ss_list_price)),count(1),channel,sales,number_sales,sum,isEmpty,count] InputAdapter Exchange [i_brand_id,i_class_id,i_category_id] #3 WholeStageCodegen (37) HashAggregate [i_brand_id,i_class_id,i_category_id,ss_quantity,ss_list_price] [sum,isEmpty,count,sum,isEmpty,count] Project [ss_quantity,ss_list_price,i_brand_id,i_class_id,i_category_id] BroadcastHashJoin [ss_item_sk,i_item_sk] Project [ss_item_sk,ss_quantity,ss_list_price] BroadcastHashJoin [ss_sold_date_sk,d_date_sk] BroadcastHashJoin [ss_item_sk,ss_item_sk] Filter [ss_item_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.store_sales [ss_item_sk,ss_quantity,ss_list_price,ss_sold_date_sk] SubqueryBroadcast [d_date_sk] #1 BroadcastExchange #4 WholeStageCodegen (1) Project [d_date_sk] Filter [d_year,d_moy,d_date_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.date_dim [d_date_sk,d_year,d_moy] InputAdapter BroadcastExchange #5 WholeStageCodegen (17) Project [i_item_sk] BroadcastHashJoin [i_brand_id,i_class_id,i_category_id,brand_id,class_id,category_id] Filter [i_brand_id,i_class_id,i_category_id] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.item [i_item_sk,i_brand_id,i_class_id,i_category_id] InputAdapter BroadcastExchange #6 WholeStageCodegen (16) SortMergeJoin [brand_id,class_id,category_id,i_brand_id,i_class_id,i_category_id] InputAdapter WholeStageCodegen (11) Sort [brand_id,class_id,category_id] InputAdapter Exchange [brand_id,class_id,category_id] #7 WholeStageCodegen (10) HashAggregate [brand_id,class_id,category_id] InputAdapter Exchange [brand_id,class_id,category_id] #8 WholeStageCodegen (9) HashAggregate [brand_id,class_id,category_id] Project [i_brand_id,i_class_id,i_category_id] BroadcastHashJoin [ss_item_sk,i_item_sk] Project [ss_item_sk] BroadcastHashJoin [ss_sold_date_sk,d_date_sk] Filter [ss_item_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.store_sales [ss_item_sk,ss_sold_date_sk] SubqueryBroadcast [d_date_sk] #2 BroadcastExchange #9 WholeStageCodegen (1) Project [d_date_sk] Filter [d_year,d_date_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.date_dim [d_date_sk,d_year] InputAdapter ReusedExchange [d_date_sk] #9 InputAdapter BroadcastExchange #10 WholeStageCodegen (8) SortMergeJoin [i_brand_id,i_class_id,i_category_id,i_brand_id,i_class_id,i_category_id] InputAdapter WholeStageCodegen (3) Sort [i_brand_id,i_class_id,i_category_id] InputAdapter Exchange [i_brand_id,i_class_id,i_category_id] #11 WholeStageCodegen (2) Filter [i_item_sk,i_brand_id,i_class_id,i_category_id] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.item [i_item_sk,i_brand_id,i_class_id,i_category_id] InputAdapter WholeStageCodegen (7) Sort [i_brand_id,i_class_id,i_category_id] InputAdapter Exchange [i_brand_id,i_class_id,i_category_id] #12 WholeStageCodegen (6) Project [i_brand_id,i_class_id,i_category_id] BroadcastHashJoin [cs_item_sk,i_item_sk] Project [cs_item_sk] BroadcastHashJoin [cs_sold_date_sk,d_date_sk] Filter [cs_item_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.catalog_sales [cs_item_sk,cs_sold_date_sk] ReusedSubquery [d_date_sk] #2 InputAdapter ReusedExchange [d_date_sk] #9 InputAdapter BroadcastExchange #13 WholeStageCodegen (5) Filter [i_item_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.item [i_item_sk,i_brand_id,i_class_id,i_category_id] InputAdapter WholeStageCodegen (15) Sort [i_brand_id,i_class_id,i_category_id] InputAdapter Exchange [i_brand_id,i_class_id,i_category_id] #14 WholeStageCodegen (14) Project [i_brand_id,i_class_id,i_category_id] BroadcastHashJoin [ws_item_sk,i_item_sk] Project [ws_item_sk] BroadcastHashJoin [ws_sold_date_sk,d_date_sk] Filter [ws_item_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.web_sales [ws_item_sk,ws_sold_date_sk] ReusedSubquery [d_date_sk] #2 InputAdapter ReusedExchange [d_date_sk] #9 InputAdapter ReusedExchange [i_item_sk,i_brand_id,i_class_id,i_category_id] #13 InputAdapter ReusedExchange [d_date_sk] #4 InputAdapter BroadcastExchange #15 WholeStageCodegen (36) BroadcastHashJoin [i_item_sk,ss_item_sk] Filter [i_item_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.item [i_item_sk,i_brand_id,i_class_id,i_category_id] InputAdapter ReusedExchange [ss_item_sk] #5 WholeStageCodegen (76) Filter [sales] ReusedSubquery [average_sales] #3 HashAggregate [i_brand_id,i_class_id,i_category_id,sum,isEmpty,count] [sum((cast(cs_quantity as decimal(10,0)) * cs_list_price)),count(1),channel,sales,number_sales,sum,isEmpty,count] InputAdapter Exchange [i_brand_id,i_class_id,i_category_id] #18 WholeStageCodegen (75) HashAggregate [i_brand_id,i_class_id,i_category_id,cs_quantity,cs_list_price] [sum,isEmpty,count,sum,isEmpty,count] Project [cs_quantity,cs_list_price,i_brand_id,i_class_id,i_category_id] BroadcastHashJoin [cs_item_sk,i_item_sk] Project [cs_item_sk,cs_quantity,cs_list_price] BroadcastHashJoin [cs_sold_date_sk,d_date_sk] BroadcastHashJoin [cs_item_sk,ss_item_sk] Filter [cs_item_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.catalog_sales [cs_item_sk,cs_quantity,cs_list_price,cs_sold_date_sk] ReusedSubquery [d_date_sk] #1 InputAdapter ReusedExchange [ss_item_sk] #5 InputAdapter ReusedExchange [d_date_sk] #4 InputAdapter ReusedExchange [i_item_sk,i_brand_id,i_class_id,i_category_id] #15 WholeStageCodegen (114) Filter [sales] ReusedSubquery [average_sales] #3 HashAggregate [i_brand_id,i_class_id,i_category_id,sum,isEmpty,count] [sum((cast(ws_quantity as decimal(10,0)) * ws_list_price)),count(1),channel,sales,number_sales,sum,isEmpty,count] InputAdapter Exchange [i_brand_id,i_class_id,i_category_id] #19 WholeStageCodegen (113) HashAggregate [i_brand_id,i_class_id,i_category_id,ws_quantity,ws_list_price] [sum,isEmpty,count,sum,isEmpty,count] Project [ws_quantity,ws_list_price,i_brand_id,i_class_id,i_category_id] BroadcastHashJoin [ws_item_sk,i_item_sk] Project [ws_item_sk,ws_quantity,ws_list_price] BroadcastHashJoin [ws_sold_date_sk,d_date_sk] BroadcastHashJoin [ws_item_sk,ss_item_sk] Filter [ws_item_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.web_sales [ws_item_sk,ws_quantity,ws_list_price,ws_sold_date_sk] ReusedSubquery [d_date_sk] #1 InputAdapter ReusedExchange [ss_item_sk] #5 InputAdapter ReusedExchange [d_date_sk] #4 InputAdapter ReusedExchange [i_item_sk,i_brand_id,i_class_id,i_category_id] #15 WholeStageCodegen (233) HashAggregate [channel,i_brand_id,i_class_id,sum,isEmpty,sum] [sum(sum_sales),sum(number_salesL),i_category_id,sum(sum_sales),sum(number_sales),sum,isEmpty,sum] InputAdapter Exchange [channel,i_brand_id,i_class_id] #20 WholeStageCodegen (232) HashAggregate [channel,i_brand_id,i_class_id,sum_sales,number_sales] [sum,isEmpty,sum,sum,isEmpty,sum] HashAggregate [channel,i_brand_id,i_class_id,i_category_id,sum,isEmpty,sum] [sum(sales),sum(number_salesL),sum_sales,number_sales,sum,isEmpty,sum] InputAdapter ReusedExchange [channel,i_brand_id,i_class_id,i_category_id,sum,isEmpty,sum] #2 WholeStageCodegen (350) HashAggregate [channel,i_brand_id,sum,isEmpty,sum] [sum(sum_sales),sum(number_salesL),i_class_id,i_category_id,sum(sum_sales),sum(number_sales),sum,isEmpty,sum] InputAdapter Exchange [channel,i_brand_id] #21 WholeStageCodegen (349) HashAggregate [channel,i_brand_id,sum_sales,number_sales] [sum,isEmpty,sum,sum,isEmpty,sum] HashAggregate [channel,i_brand_id,i_class_id,i_category_id,sum,isEmpty,sum] [sum(sales),sum(number_salesL),sum_sales,number_sales,sum,isEmpty,sum] InputAdapter ReusedExchange [channel,i_brand_id,i_class_id,i_category_id,sum,isEmpty,sum] #2 WholeStageCodegen (467) HashAggregate [channel,sum,isEmpty,sum] [sum(sum_sales),sum(number_salesL),i_brand_id,i_class_id,i_category_id,sum(sum_sales),sum(number_sales),sum,isEmpty,sum] InputAdapter Exchange [channel] #22 WholeStageCodegen (466) HashAggregate [channel,sum_sales,number_sales] [sum,isEmpty,sum,sum,isEmpty,sum] HashAggregate [channel,i_brand_id,i_class_id,i_category_id,sum,isEmpty,sum] [sum(sales),sum(number_salesL),sum_sales,number_sales,sum,isEmpty,sum] InputAdapter ReusedExchange [channel,i_brand_id,i_class_id,i_category_id,sum,isEmpty,sum] #2 WholeStageCodegen (584) HashAggregate [sum,isEmpty,sum] [sum(sum_sales),sum(number_salesL),channel,i_brand_id,i_class_id,i_category_id,sum(sum_sales),sum(number_sales),sum,isEmpty,sum] InputAdapter Exchange #23 WholeStageCodegen (583) HashAggregate [sum_sales,number_sales] [sum,isEmpty,sum,sum,isEmpty,sum] HashAggregate [channel,i_brand_id,i_class_id,i_category_id,sum,isEmpty,sum] [sum(sales),sum(number_salesL),sum_sales,number_sales,sum,isEmpty,sum] InputAdapter ReusedExchange [channel,i_brand_id,i_class_id,i_category_id,sum,isEmpty,sum] #2 actual simplified plan: /home/runner/work/spark/spark/target/tmp/q14a.sf100.actual.simplified.txt actual explain plan: /home/runner/work/spark/spark/target/tmp/q14a.sf100.actual.explain.txt TakeOrderedAndProject [channel,i_brand_id,i_class_id,i_category_id,sum_sales,number_sales] WholeStageCodegen (586) HashAggregate [channel,i_brand_id,i_class_id,i_category_id,sum_sales,number_sales] InputAdapter Exchange [channel,i_brand_id,i_class_id,i_category_id,sum_sales,number_sales] #1 WholeStageCodegen (585) HashAggregate [channel,i_brand_id,i_class_id,i_category_id,sum_sales,number_sales] InputAdapter Union WholeStageCodegen (116) HashAggregate [channel,i_brand_id,i_class_id,i_category_id,sum,isEmpty,sum] [sum(sales),sum(number_salesL),sum_sales,number_sales,sum,isEmpty,sum] InputAdapter Exchange [channel,i_brand_id,i_class_id,i_category_id] #2 WholeStageCodegen (115) HashAggregate [channel,i_brand_id,i_class_id,i_category_id,sales,number_sales] [sum,isEmpty,sum,sum,isEmpty,sum] InputAdapter Union WholeStageCodegen (38) Filter [sales] Subquery #4 WholeStageCodegen (8) HashAggregate [sum,count] [avg((cast(quantity as decimal(10,0)) * list_price)),average_sales,sum,count] InputAdapter Exchange #17 WholeStageCodegen (7) HashAggregate [quantity,list_price] [sum,count,sum,count] InputAdapter Union WholeStageCodegen (2) Project [ss_quantity,ss_list_price] BroadcastHashJoin [ss_sold_date_sk,d_date_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.store_sales [ss_quantity,ss_list_price,ss_sold_date_sk] ReusedSubquery [d_date_sk] #2 InputAdapter ReusedExchange [d_date_sk] #9 WholeStageCodegen (4) Project [cs_quantity,cs_list_price] BroadcastHashJoin [cs_sold_date_sk,d_date_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.catalog_sales [cs_quantity,cs_list_price,cs_sold_date_sk] SubqueryBroadcast [d_date_sk] #5 BroadcastExchange #18 WholeStageCodegen (1) Project [d_date_sk] Filter [d_year,d_date_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.date_dim [d_date_sk,d_year] InputAdapter ReusedExchange [d_date_sk] #18 WholeStageCodegen (6) Project [ws_quantity,ws_list_price] BroadcastHashJoin [ws_sold_date_sk,d_date_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.web_sales [ws_quantity,ws_list_price,ws_sold_date_sk] ReusedSubquery [d_date_sk] #5 InputAdapter ReusedExchange [d_date_sk] #18 HashAggregate [i_brand_id,i_class_id,i_category_id,sum,isEmpty,count] [sum((cast(ss_quantity as decimal(10,0)) * ss_list_price)),count(1),channel,sales,number_sales,sum,isEmpty,count] InputAdapter Exchange [i_brand_id,i_class_id,i_category_id] #3 WholeStageCodegen (37) HashAggregate [i_brand_id,i_class_id,i_category_id,ss_quantity,ss_list_price] [sum,isEmpty,count,sum,isEmpty,count] Project [ss_quantity,ss_list_price,i_brand_id,i_class_id,i_category_id] BroadcastHashJoin [ss_item_sk,i_item_sk] Project [ss_item_sk,ss_quantity,ss_list_price] BroadcastHashJoin [ss_sold_date_sk,d_date_sk] BroadcastHashJoin [ss_item_sk,ss_item_sk] Filter [ss_item_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.store_sales [ss_item_sk,ss_quantity,ss_list_price,ss_sold_date_sk] SubqueryBroadcast [d_date_sk] #1 BroadcastExchange #4 WholeStageCodegen (1) Project [d_date_sk] Filter [d_year,d_moy,d_date_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.date_dim [d_date_sk,d_year,d_moy] InputAdapter BroadcastExchange #5 WholeStageCodegen (17) Project [i_item_sk] BroadcastHashJoin [i_brand_id,i_class_id,i_category_id,brand_id,class_id,category_id] Filter [i_brand_id,i_class_id,i_category_id] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.item [i_item_sk,i_brand_id,i_class_id,i_category_id] InputAdapter BroadcastExchange #6 WholeStageCodegen (16) SortMergeJoin [brand_id,class_id,category_id,i_brand_id,i_class_id,i_category_id] InputAdapter WholeStageCodegen (11) Sort [brand_id,class_id,category_id] InputAdapter Exchange [brand_id,class_id,category_id] #7 WholeStageCodegen (10) HashAggregate [brand_id,class_id,category_id] InputAdapter Exchange [brand_id,class_id,category_id] #8 WholeStageCodegen (9) HashAggregate [brand_id,class_id,category_id] Project [i_brand_id,i_class_id,i_category_id] BroadcastHashJoin [ss_item_sk,i_item_sk] Project [ss_item_sk] BroadcastHashJoin [ss_sold_date_sk,d_date_sk] Filter [ss_item_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.store_sales [ss_item_sk,ss_sold_date_sk] SubqueryBroadcast [d_date_sk] #2 BroadcastExchange #9 WholeStageCodegen (1) Project [d_date_sk] Filter [d_year,d_date_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.date_dim [d_date_sk,d_year] InputAdapter ReusedExchange [d_date_sk] #9 InputAdapter BroadcastExchange #10 WholeStageCodegen (8) SortMergeJoin [i_brand_id,i_class_id,i_category_id,i_brand_id,i_class_id,i_category_id] InputAdapter WholeStageCodegen (3) Sort [i_brand_id,i_class_id,i_category_id] InputAdapter Exchange [i_brand_id,i_class_id,i_category_id] #11 WholeStageCodegen (2) Filter [i_item_sk,i_brand_id,i_class_id,i_category_id] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.item [i_item_sk,i_brand_id,i_class_id,i_category_id] InputAdapter WholeStageCodegen (7) Sort [i_brand_id,i_class_id,i_category_id] InputAdapter Exchange [i_brand_id,i_class_id,i_category_id] #12 WholeStageCodegen (6) Project [i_brand_id,i_class_id,i_category_id] BroadcastHashJoin [cs_item_sk,i_item_sk] Project [cs_item_sk] BroadcastHashJoin [cs_sold_date_sk,d_date_sk] Filter [cs_item_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.catalog_sales [cs_item_sk,cs_sold_date_sk] ReusedSubquery [d_date_sk] #2 InputAdapter ReusedExchange [d_date_sk] #9 InputAdapter BroadcastExchange #13 WholeStageCodegen (5) Filter [i_item_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.item [i_item_sk,i_brand_id,i_class_id,i_category_id] InputAdapter WholeStageCodegen (15) Sort [i_brand_id,i_class_id,i_category_id] InputAdapter Exchange [i_brand_id,i_class_id,i_category_id] #14 WholeStageCodegen (14) Project [i_brand_id,i_class_id,i_category_id] BroadcastHashJoin [ws_item_sk,i_item_sk] Project [ws_item_sk] BroadcastHashJoin [ws_sold_date_sk,d_date_sk] Filter [ws_item_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.web_sales [ws_item_sk,ws_sold_date_sk] ReusedSubquery [d_date_sk] #2 InputAdapter ReusedExchange [d_date_sk] #9 InputAdapter ReusedExchange [i_item_sk,i_brand_id,i_class_id,i_category_id] #13 InputAdapter ReusedExchange [d_date_sk] #4 InputAdapter BroadcastExchange #15 WholeStageCodegen (36) BroadcastHashJoin [i_item_sk,ss_item_sk] Filter [i_item_sk] Subquery #3 ObjectHashAggregate [buf] [bloom_filter_agg(xxhash64(ss_item_sk, 42), 5012524, 36583609, 0, 0),bloomFilter,buf] Exchange #16 ObjectHashAggregate [ss_item_sk] [buf,buf] WholeStageCodegen (19) Project [ss_item_sk] BroadcastHashJoin [ss_sold_date_sk,d_date_sk] Project [ss_item_sk,ss_sold_date_sk] Filter [ss_sold_date_sk] ReusedSubquery [d_date_sk] #1 BroadcastHashJoin [ss_item_sk,ss_item_sk] Filter [ss_item_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.store_sales [ss_item_sk,ss_sold_date_sk] InputAdapter ReusedExchange [ss_item_sk] #5 InputAdapter ReusedExchange [d_date_sk] #4 ColumnarToRow InputAdapter Scan parquet spark_catalog.default.item [i_item_sk,i_brand_id,i_class_id,i_category_id] InputAdapter ReusedExchange [ss_item_sk] #5 WholeStageCodegen (76) Filter [sales] ReusedSubquery [average_sales] #4 HashAggregate [i_brand_id,i_class_id,i_category_id,sum,isEmpty,count] [sum((cast(cs_quantity as decimal(10,0)) * cs_list_price)),count(1),channel,sales,number_sales,sum,isEmpty,count] InputAdapter Exchange [i_brand_id,i_class_id,i_category_id] #19 WholeStageCodegen (75) HashAggregate [i_brand_id,i_class_id,i_category_id,cs_quantity,cs_list_price] [sum,isEmpty,count,sum,isEmpty,count] Project [cs_quantity,cs_list_price,i_brand_id,i_class_id,i_category_id] BroadcastHashJoin [cs_item_sk,i_item_sk] Project [cs_item_sk,cs_quantity,cs_list_price] BroadcastHashJoin [cs_sold_date_sk,d_date_sk] BroadcastHashJoin [cs_item_sk,ss_item_sk] Filter [cs_item_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.catalog_sales [cs_item_sk,cs_quantity,cs_list_price,cs_sold_date_sk] ReusedSubquery [d_date_sk] #1 InputAdapter ReusedExchange [ss_item_sk] #5 InputAdapter ReusedExchange [d_date_sk] #4 InputAdapter BroadcastExchange #20 WholeStageCodegen (74) BroadcastHashJoin [i_item_sk,ss_item_sk] Filter [i_item_sk] Subquery #6 ObjectHashAggregate [buf] [bloom_filter_agg(xxhash64(cs_item_sk, 42), 2517915, 20802585, 0, 0),bloomFilter,buf] Exchange #21 ObjectHashAggregate [cs_item_sk] [buf,buf] WholeStageCodegen (19) Project [cs_item_sk] BroadcastHashJoin [cs_sold_date_sk,d_date_sk] Project [cs_item_sk,cs_sold_date_sk] Filter [cs_sold_date_sk] ReusedSubquery [d_date_sk] #1 BroadcastHashJoin [cs_item_sk,ss_item_sk] Filter [cs_item_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.catalog_sales [cs_item_sk,cs_sold_date_sk] InputAdapter ReusedExchange [ss_item_sk] #5 InputAdapter ReusedExchange [d_date_sk] #4 ColumnarToRow InputAdapter Scan parquet spark_catalog.default.item [i_item_sk,i_brand_id,i_class_id,i_category_id] InputAdapter ReusedExchange [ss_item_sk] #5 WholeStageCodegen (114) Filter [sales] ReusedSubquery [average_sales] #4 HashAggregate [i_brand_id,i_class_id,i_category_id,sum,isEmpty,count] [sum((cast(ws_quantity as decimal(10,0)) * ws_list_price)),count(1),channel,sales,number_sales,sum,isEmpty,count] InputAdapter Exchange [i_brand_id,i_class_id,i_category_id] #22 WholeStageCodegen (113) HashAggregate [i_brand_id,i_class_id,i_category_id,ws_quantity,ws_list_price] [sum,isEmpty,count,sum,isEmpty,count] Project [ws_quantity,ws_list_price,i_brand_id,i_class_id,i_category_id] BroadcastHashJoin [ws_item_sk,i_item_sk] Project [ws_item_sk,ws_quantity,ws_list_price] BroadcastHashJoin [ws_sold_date_sk,d_date_sk] BroadcastHashJoin [ws_item_sk,ss_item_sk] Filter [ws_item_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.web_sales [ws_item_sk,ws_quantity,ws_list_price,ws_sold_date_sk] ReusedSubquery [d_date_sk] #1 InputAdapter ReusedExchange [ss_item_sk] #5 InputAdapter ReusedExchange [d_date_sk] #4 InputAdapter BroadcastExchange #23 WholeStageCodegen (112) BroadcastHashJoin [i_item_sk,ss_item_sk] Filter [i_item_sk] Subquery #7 ObjectHashAggregate [buf] [bloom_filter_agg(xxhash64(ws_item_sk, 42), 1252939, 12171674, 0, 0),bloomFilter,buf] Exchange #24 ObjectHashAggregate [ws_item_sk] [buf,buf] WholeStageCodegen (19) Project [ws_item_sk] BroadcastHashJoin [ws_sold_date_sk,d_date_sk] Project [ws_item_sk,ws_sold_date_sk] Filter [ws_sold_date_sk] ReusedSubquery [d_date_sk] #1 BroadcastHashJoin [ws_item_sk,ss_item_sk] Filter [ws_item_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.web_sales [ws_item_sk,ws_sold_date_sk] InputAdapter ReusedExchange [ss_item_sk] #5 InputAdapter ReusedExchange [d_date_sk] #4 ColumnarToRow InputAdapter Scan parquet spark_catalog.default.item [i_item_sk,i_brand_id,i_class_id,i_category_id] InputAdapter ReusedExchange [ss_item_sk] #5 WholeStageCodegen (233) HashAggregate [channel,i_brand_id,i_class_id,sum,isEmpty,sum] [sum(sum_sales),sum(number_salesL),i_category_id,sum(sum_sales),sum(number_sales),sum,isEmpty,sum] InputAdapter Exchange [channel,i_brand_id,i_class_id] #25 WholeStageCodegen (232) HashAggregate [channel,i_brand_id,i_class_id,sum_sales,number_sales] [sum,isEmpty,sum,sum,isEmpty,sum] HashAggregate [channel,i_brand_id,i_class_id,i_category_id,sum,isEmpty,sum] [sum(sales),sum(number_salesL),sum_sales,number_sales,sum,isEmpty,sum] InputAdapter ReusedExchange [channel,i_brand_id,i_class_id,i_category_id,sum,isEmpty,sum] #2 WholeStageCodegen (350) HashAggregate [channel,i_brand_id,sum,isEmpty,sum] [sum(sum_sales),sum(number_salesL),i_class_id,i_category_id,sum(sum_sales),sum(number_sales),sum,isEmpty,sum] InputAdapter Exchange [channel,i_brand_id] #26 WholeStageCodegen (349) HashAggregate [channel,i_brand_id,sum_sales,number_sales] [sum,isEmpty,sum,sum,isEmpty,sum] HashAggregate [channel,i_brand_id,i_clas
TPCDSV2_7_PlanStabilityWithStatsSuite.check simplified sf100 (tpcds-v2.7.0/q70a): TPCDSV2_7_PlanStabilityWithStatsSuite#L316
org.scalatest.exceptions.TestFailedException: Plans did not match: last approved simplified plan: /home/runner/work/spark/spark/sql/core/src/test/resources/tpcds-plan-stability/approved-plans-v2_7/q70a.sf100/simplified.txt last approved explain plan: /home/runner/work/spark/spark/sql/core/src/test/resources/tpcds-plan-stability/approved-plans-v2_7/q70a.sf100/explain.txt TakeOrderedAndProject [lochierarchy,s_state,rank_within_parent,total_sum,s_county] WholeStageCodegen (33) Project [total_sum,s_state,s_county,lochierarchy,rank_within_parent] InputAdapter Window [total_sum,lochierarchy,_w0] WholeStageCodegen (32) Sort [lochierarchy,_w0,total_sum] InputAdapter Exchange [lochierarchy,_w0] #1 WholeStageCodegen (31) HashAggregate [total_sum,s_state,s_county,g_state,g_county,lochierarchy] [_w0] InputAdapter Exchange [total_sum,s_state,s_county,g_state,g_county,lochierarchy] #2 WholeStageCodegen (30) HashAggregate [total_sum,s_state,s_county,g_state,g_county,lochierarchy] InputAdapter Union WholeStageCodegen (9) HashAggregate [s_state,s_county,sum] [sum(UnscaledValue(ss_net_profit)),total_sum,g_state,g_county,lochierarchy,sum] InputAdapter Exchange [s_state,s_county] #3 WholeStageCodegen (8) HashAggregate [s_state,s_county,ss_net_profit] [sum,sum] Project [ss_net_profit,s_county,s_state] BroadcastHashJoin [ss_store_sk,s_store_sk] Project [ss_store_sk,ss_net_profit] BroadcastHashJoin [ss_sold_date_sk,d_date_sk] Filter [ss_store_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.store_sales [ss_store_sk,ss_net_profit,ss_sold_date_sk] SubqueryBroadcast [d_date_sk] #1 BroadcastExchange #4 WholeStageCodegen (1) Project [d_date_sk] Filter [d_month_seq,d_date_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.date_dim [d_date_sk,d_month_seq] InputAdapter ReusedExchange [d_date_sk] #4 InputAdapter BroadcastExchange #5 WholeStageCodegen (7) BroadcastHashJoin [s_state,s_state] Filter [s_store_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.store [s_store_sk,s_county,s_state] InputAdapter BroadcastExchange #6 WholeStageCodegen (6) Project [s_state] Filter [ranking] InputAdapter Window [_w0,s_state] WindowGroupLimit [s_state,_w0] WholeStageCodegen (5) Sort [s_state,_w0] HashAggregate [sum] [sum(UnscaledValue(ss_net_profit)),_w0,s_state,sum] InputAdapter Exchange [s_state] #7 WholeStageCodegen (4) HashAggregate [s_state,ss_net_profit] [sum,sum] Project [ss_net_profit,s_state] BroadcastHashJoin [ss_store_sk,s_store_sk] Project [ss_store_sk,ss_net_profit] BroadcastHashJoin [ss_sold_date_sk,d_date_sk] Filter [ss_store_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.store_sales [ss_store_sk,ss_net_profit,ss_sold_date_sk] ReusedSubquery [d_date_sk] #1 InputAdapter ReusedExchange [d_date_sk] #4 InputAdapter BroadcastExchange #8 WholeStageCodegen (3) Filter [s_store_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.store [s_store_sk,s_state] WholeStageCodegen (19) HashAggregate [s_state,sum,isEmpty] [sum(total_sum),total_sum,s_county,g_state,g_county,lochierarchy,sum,isEmpty] InputAdapter Exchange [s_state] #9 WholeStageCodegen (18) HashAggregate [s_state,total_sum] [sum,isEmpty,sum,isEmpty] HashAggregate [s_state,s_county,sum] [sum(UnscaledValue(ss_net_profit)),total_sum,sum] InputAdapter ReusedExchange [s_state,s_county,sum] #3 WholeStageCodegen (29) HashAggregate [sum,isEmpty] [sum(total_sum),total_sum,s_state,s_county,g_state,g_county,lochierarchy,sum,isEmpty] InputAdapter Exchange #10 WholeStageCodegen (28) HashAggregate [total_sum] [sum,isEmpty,sum,isEmpty] HashAggregate [s_state,s_county,sum] [sum(UnscaledValue(ss_net_profit)),total_sum,sum] InputAdapter ReusedExchange [s_state,s_county,sum] #3 actual simplified plan: /home/runner/work/spark/spark/target/tmp/q70a.sf100.actual.simplified.txt actual explain plan: /home/runner/work/spark/spark/target/tmp/q70a.sf100.actual.explain.txt TakeOrderedAndProject [lochierarchy,s_state,rank_within_parent,total_sum,s_county] WholeStageCodegen (33) Project [total_sum,s_state,s_county,lochierarchy,rank_within_parent] InputAdapter Window [total_sum,lochierarchy,_w0] WholeStageCodegen (32) Sort [lochierarchy,_w0,total_sum] InputAdapter Exchange [lochierarchy,_w0] #1 WholeStageCodegen (31) HashAggregate [total_sum,s_state,s_county,g_state,g_county,lochierarchy] [_w0] InputAdapter Exchange [total_sum,s_state,s_county,g_state,g_county,lochierarchy] #2 WholeStageCodegen (30) HashAggregate [total_sum,s_state,s_county,g_state,g_county,lochierarchy] InputAdapter Union WholeStageCodegen (9) HashAggregate [s_state,s_county,sum] [sum(UnscaledValue(ss_net_profit)),total_sum,g_state,g_county,lochierarchy,sum] InputAdapter Exchange [s_state,s_county] #3 WholeStageCodegen (8) HashAggregate [s_state,s_county,ss_net_profit] [sum,sum] Project [ss_net_profit,s_county,s_state] BroadcastHashJoin [ss_store_sk,s_store_sk] Project [ss_store_sk,ss_net_profit] BroadcastHashJoin [ss_sold_date_sk,d_date_sk] Filter [ss_store_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.store_sales [ss_store_sk,ss_net_profit,ss_sold_date_sk] SubqueryBroadcast [d_date_sk] #1 BroadcastExchange #4 WholeStageCodegen (1) Project [d_date_sk] Filter [d_month_seq,d_date_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.date_dim [d_date_sk,d_month_seq] InputAdapter ReusedExchange [d_date_sk] #4 InputAdapter BroadcastExchange #5 WholeStageCodegen (7) BroadcastHashJoin [s_state,s_state] Filter [s_store_sk] Subquery #2 ObjectHashAggregate [buf] [bloom_filter_agg(xxhash64(ss_store_sk, 42), 54151237, 67108864, 0, 0),bloomFilter,buf] Exchange #6 ObjectHashAggregate [ss_store_sk] [buf,buf] WholeStageCodegen (2) Project [ss_store_sk] BroadcastHashJoin [ss_sold_date_sk,d_date_sk] Filter [ss_store_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.store_sales [ss_store_sk,ss_sold_date_sk] ReusedSubquery [d_date_sk] #1 InputAdapter ReusedExchange [d_date_sk] #4 ColumnarToRow InputAdapter Scan parquet spark_catalog.default.store [s_store_sk,s_county,s_state] InputAdapter BroadcastExchange #7 WholeStageCodegen (6) Project [s_state] Filter [ranking] InputAdapter Window [_w0,s_state] WindowGroupLimit [s_state,_w0] WholeStageCodegen (5) Sort [s_state,_w0] HashAggregate [sum] [sum(UnscaledValue(ss_net_profit)),_w0,s_state,sum] InputAdapter Exchange [s_state] #8 WholeStageCodegen (4) HashAggregate [s_state,ss_net_profit] [sum,sum] Project [ss_net_profit,s_state] BroadcastHashJoin [ss_store_sk,s_store_sk] Project [ss_store_sk,ss_net_profit] BroadcastHashJoin [ss_sold_date_sk,d_date_sk] Filter [ss_store_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.store_sales [ss_store_sk,ss_net_profit,ss_sold_date_sk] ReusedSubquery [d_date_sk] #1 InputAdapter ReusedExchange [d_date_sk] #4 InputAdapter BroadcastExchange #9 WholeStageCodegen (3) Filter [s_store_sk] ColumnarToRow InputAdapter Scan parquet spark_catalog.default.store [s_store_sk,s_state] WholeStageCodegen (19) HashAggregate [s_state,sum,isEmpty] [sum(total_sum),total_sum,s_county,g_state,g_county,lochierarchy,sum,isEmpty] InputAdapter Exchange [s_state] #10 WholeStageCodegen (18) HashAggregate [s_state,total_sum] [sum,isEmpty,sum,isEmpty] HashAggregate [s_state,s_county,sum] [sum(UnscaledValue(ss_net_profit)),total_sum,sum] InputAdapter ReusedExchange [s_state,s_county,sum] #3 WholeStageCodegen (29) HashAggregate [sum,isEmpty] [sum(total_sum),total_sum,s_state,s_county,g_state,g_county,lochierarchy,sum,isEmpty] InputAdapter Exchange #11 WholeStageCodegen (28) HashAggregate [total_sum] [sum,isEmpty,sum,isEmpty] HashAggregate [s_state,s_county,sum] [sum(UnscaledValue(ss_net_profit)),total_sum,sum] InputAdapter ReusedExchange [s_state,s_county,sum] #3
Run / Build modules: pyspark-core, pyspark-streaming
No files were found with the provided path: **/target/test-reports/*.xml. No artifacts will be uploaded.
Run / Build modules: pyspark-errors
No files were found with the provided path: **/target/test-reports/*.xml. No artifacts will be uploaded.

Artifacts

Produced during runtime
Name Size
test-results-catalyst, hive-thriftserver--8-hadoop3-hive2.3 Expired
3.01 MB
test-results-core, unsafe, kvstore, avro, network-common, network-shuffle, repl, launcher, examples, sketch, graphx--8-hadoop3-hive2.3 Expired
154 KB
test-results-docker-integration--8-hadoop3-hive2.3 Expired
133 KB
test-results-hive-- other tests-8-hadoop3-hive2.3 Expired
1.1 MB
test-results-hive-- slow tests-8-hadoop3-hive2.3 Expired
943 KB
test-results-pyspark-connect--8-hadoop3-hive2.3 Expired
319 KB
test-results-pyspark-mllib, pyspark-ml, pyspark-ml-connect--8-hadoop3-hive2.3 Expired
424 KB
test-results-pyspark-pandas--8-hadoop3-hive2.3 Expired
1.06 MB
test-results-pyspark-pandas-connect--8-hadoop3-hive2.3 Expired
1.44 MB
test-results-pyspark-pandas-slow--8-hadoop3-hive2.3 Expired
1.27 MB
test-results-pyspark-pandas-slow-connect--8-hadoop3-hive2.3 Expired
745 KB
test-results-pyspark-sql, pyspark-resource, pyspark-testing--8-hadoop3-hive2.3 Expired
360 KB
test-results-sparkr--8-hadoop3-hive2.3 Expired
280 KB
test-results-sql-- extended tests-8-hadoop3-hive2.3 Expired
4.3 MB
test-results-sql-- other tests-8-hadoop3-hive2.3 Expired
4.66 MB
test-results-sql-- slow tests-8-hadoop3-hive2.3 Expired
3.17 MB
test-results-streaming, sql-kafka-0-10, streaming-kafka-0-10, mllib-local, mllib, yarn, mesos, kubernetes, hadoop-cloud, spark-ganglia-lgpl, connect, protobuf--8-hadoop3-hive2.3 Expired
2.33 MB
test-results-tpcds--8-hadoop3-hive2.3 Expired
22.6 KB
unit-tests-log-sql-- extended tests-8-hadoop3-hive2.3 Expired
512 MB