Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[regression test](routine test) add case for desired_concurrent_number #27372

Merged
merged 3 commits into from
Nov 23, 2023
Merged

[regression test](routine test) add case for desired_concurrent_number #27372

merged 3 commits into from
Nov 23, 2023

Conversation

liugddx
Copy link
Member

@liugddx liugddx commented Nov 21, 2023

Proposed changes

Issue Number: close #xxx

Further comments

If this is a relatively large or complex change, kick off the discussion at [email protected] by explaining why you chose the solution you did and what alternatives you considered, etc...

@liugddx
Copy link
Member Author

liugddx commented Nov 21, 2023

run buildall

@doris-robot
Copy link

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G' 
Tpch sf100 test result on commit 7873f9a73ed368a5a83dc0fa333df83e1032b2ee, data reload: false

run tpch-sf100 query with default conf and session variables
q1	4889	4672	4648	4648
q2	358	145	158	145
q3	2029	1917	1956	1917
q4	1374	1232	1260	1232
q5	3943	3896	3996	3896
q6	253	135	130	130
q7	1434	878	889	878
q8	2751	2767	2752	2752
q9	9813	9549	9560	9549
q10	3463	3543	3492	3492
q11	376	250	250	250
q12	441	289	294	289
q13	4577	3833	3786	3786
q14	319	287	283	283
q15	584	538	527	527
q16	671	571	582	571
q17	1126	964	921	921
q18	7718	7362	7429	7362
q19	1650	1668	1678	1668
q20	541	328	299	299
q21	4405	3960	3917	3917
q22	475	379	369	369
Total cold run time: 53190 ms
Total hot run time: 48881 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	4609	4572	4574	4572
q2	353	241	270	241
q3	4016	3996	3990	3990
q4	2700	2681	2697	2681
q5	9766	9619	9669	9619
q6	246	125	126	125
q7	2986	2490	2467	2467
q8	4425	4465	4424	4424
q9	13296	13126	13108	13108
q10	4087	4198	4198	4198
q11	806	643	647	643
q12	976	812	799	799
q13	4273	3575	3591	3575
q14	383	353	354	353
q15	569	529	516	516
q16	727	691	683	683
q17	3830	3947	3898	3898
q18	9574	8919	8961	8919
q19	1777	1783	1791	1783
q20	2388	2076	2064	2064
q21	8729	8389	8629	8389
q22	884	834	780	780
Total cold run time: 81400 ms
Total hot run time: 77827 ms

@doris-robot
Copy link

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.51 seconds
stream load tsv: 577 seconds loaded 74807831229 Bytes, about 123 MB/s
stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 34 seconds loaded 861443392 Bytes, about 24 MB/s
insert into select: 28.7 seconds inserted 10000000 Rows, about 348K ops/s
storage size: 17100903243 Bytes

@liugddx
Copy link
Member Author

liugddx commented Nov 22, 2023

run P0

@liugddx
Copy link
Member Author

liugddx commented Nov 22, 2023

run pipelinex_P0

@liugddx
Copy link
Member Author

liugddx commented Nov 22, 2023

run pipelinex_p0

@liugddx
Copy link
Member Author

liugddx commented Nov 22, 2023

run p0

@liugddx
Copy link
Member Author

liugddx commented Nov 22, 2023

run pipelinex_p0

@liugddx
Copy link
Member Author

liugddx commented Nov 22, 2023

run buildall

@doris-robot
Copy link

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G' 
Tpch sf100 test result on commit 83c2fe7617b040f1b1a283f27c8b5abf89aaeddd, data reload: false

run tpch-sf100 query with default conf and session variables
q1	4885	4656	4643	4643
q2	355	157	159	157
q3	2029	1917	1881	1881
q4	1414	1270	1239	1239
q5	3978	3967	4003	3967
q6	246	125	133	125
q7	1421	903	919	903
q8	2758	2769	2751	2751
q9	9781	9510	9669	9510
q10	3473	3545	3518	3518
q11	375	245	244	244
q12	441	291	287	287
q13	4571	3805	3823	3805
q14	315	279	296	279
q15	590	539	523	523
q16	660	593	587	587
q17	1124	960	953	953
q18	7815	7382	7450	7382
q19	1668	1693	1705	1693
q20	536	314	293	293
q21	4328	3954	3964	3954
q22	481	379	370	370
Total cold run time: 53244 ms
Total hot run time: 49064 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	4586	4544	4568	4544
q2	355	235	275	235
q3	3997	4000	3987	3987
q4	2717	2689	2689	2689
q5	9674	9649	9697	9649
q6	243	130	126	126
q7	3020	2477	2521	2477
q8	4448	4423	4401	4401
q9	13247	13139	13141	13139
q10	4086	4188	4212	4188
q11	786	688	634	634
q12	977	799	803	799
q13	4279	3588	3572	3572
q14	381	343	339	339
q15	577	518	517	517
q16	733	675	684	675
q17	3847	3841	3862	3841
q18	9509	9042	9005	9005
q19	1809	1768	1781	1768
q20	2391	2084	2073	2073
q21	8786	8696	8623	8623
q22	926	818	837	818
Total cold run time: 81374 ms
Total hot run time: 78099 ms

@doris-robot
Copy link

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 44.74 seconds
stream load tsv: 566 seconds loaded 74807831229 Bytes, about 126 MB/s
stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.5 seconds inserted 10000000 Rows, about 350K ops/s
storage size: 17099496671 Bytes

@liugddx
Copy link
Member Author

liugddx commented Nov 22, 2023

run pipelinex_p0

@liugddx
Copy link
Member Author

liugddx commented Nov 22, 2023

run p0

@liugddx
Copy link
Member Author

liugddx commented Nov 22, 2023

run pipelinex_p0

@liugddx
Copy link
Member Author

liugddx commented Nov 22, 2023

run pipelinex_p0

sollhui
sollhui reviewed Nov 23, 2023
View reviewed changes
regression-test/suites/load_p0/routine_load/test_routine_load.groovy
COLUMNS TERMINATED BY "|"
PROPERTIES
(
"desired_concurrent_number" = "3",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

need check desired_concurrent_number value

@liugddx
Copy link
Member Author

liugddx commented Nov 23, 2023

run buildall

sollhui
sollhui approved these changes Nov 23, 2023
View reviewed changes
Copy link
Contributor

@sollhui sollhui left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

dataroaring
dataroaring approved these changes Nov 23, 2023
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Nov 23, 2023
@doris-robot
Copy link

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G' 
Tpch sf100 test result on commit 5ab6c7cdc75969c0d6a5a3ac608ba9df98236845, data reload: false

run tpch-sf100 query with default conf and session variables
q1	4930	4655	4684	4655
q2	359	166	156	156
q3	2042	1913	1950	1913
q4	1391	1281	1268	1268
q5	3998	3937	4032	3937
q6	256	129	139	129
q7	1419	885	903	885
q8	2786	2786	2771	2771
q9	9583	9653	9760	9653
q10	3452	3509	3489	3489
q11	372	256	247	247
q12	438	293	291	291
q13	4566	3834	3833	3833
q14	326	295	294	294
q15	591	550	520	520
q16	669	592	583	583
q17	1144	945	906	906
q18	7765	7345	7491	7345
q19	1675	1684	1651	1651
q20	548	302	294	294
q21	4428	4039	3999	3999
q22	478	374	382	374
Total cold run time: 53216 ms
Total hot run time: 49193 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	4588	4567	4569	4567
q2	334	246	255	246
q3	4024	4005	3987	3987
q4	2696	2710	2699	2699
q5	9561	9530	9541	9530
q6	247	124	126	124
q7	3019	2480	2483	2480
q8	4454	4464	4415	4415
q9	12929	12882	12797	12797
q10	4065	4120	4169	4120
q11	816	623	630	623
q12	971	813	814	813
q13	4300	3624	3542	3542
q14	387	366	345	345
q15	570	522	524	522
q16	762	688	685	685
q17	3928	3887	3906	3887
q18	9539	9029	9024	9024
q19	1880	1762	1768	1762
q20	2399	2054	2074	2054
q21	8820	8657	8726	8657
q22	840	772	778	772
Total cold run time: 81129 ms
Total hot run time: 77651 ms

@dataroaring dataroaring merged commit c884e46 into apache:master Nov 23, 2023
@doris-robot
Copy link

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 44.2 seconds
stream load tsv: 585 seconds loaded 74807831229 Bytes, about 121 MB/s
stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 34 seconds loaded 861443392 Bytes, about 24 MB/s
insert into select: 28.9 seconds inserted 10000000 Rows, about 346K ops/s
storage size: 17099886288 Bytes

seawinde pushed a commit to seawinde/doris that referenced this pull request Nov 28, 2023
@liugddx @seawinde
[regression test](routine test) add case for desired_concurrent_number (
9dda3ff 
apache#27372)
XuJianxu pushed a commit to XuJianxu/doris that referenced this pull request Dec 14, 2023
@liugddx
[regression test](routine test) add case for desired_concurrent_number (
75cfaa7 
apache#27372)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@sollhui sollhui sollhui approved these changes

@dataroaring dataroaring dataroaring approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/2.0.4 reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@liugddx @doris-robot @sollhui @dataroaring @xiaokang