Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Improvement](tablet) Prefetch tablets before opening #47501

Merged
merged 6 commits into from
Feb 7, 2025
Merged

[Improvement](tablet) Prefetch tablets before opening #47501

merged 6 commits into from
Feb 7, 2025

Conversation

Gabriel39
Copy link
Contributor

@Gabriel39 Gabriel39 commented Feb 5, 2025 •
edited
Loading

What problem does this PR solve?

Release note

For a scan operator in a query, FE will assign a tablet id to a specific scan operator and then it will be search by storage engine during execution. But this searching process will be done after runtime filter reached. However, if we wait for a runtime filter for a long time and compaction / balance tasks is done at the same time, a tablet / rowset will be lost and we got an error.

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@Thearas
Copy link
Contributor

Thearas commented Feb 5, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@Gabriel39
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 42.08% (11001/26145)
Line Coverage: 32.35% (92867/287051)
Region Coverage: 31.50% (47615/151158)
Branch Coverage: 27.53% (24096/87540)
Coverage Report: http://coverage.selectdb-in.cc/coverage/ff7439ab7dbc40e8cd3525f473b29cad01000c27_ff7439ab7dbc40e8cd3525f473b29cad01000c27/report/index.html

@Gabriel39
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 32511 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 7f1b3b74f77c9402509a65a343eac3e4ed7cf63c, data reload: false

------ Round 1 ----------------------------------
q1	17604	6403	5438	5438
q2	2048	314	175	175
q3	10399	1273	724	724
q4	10223	987	548	548
q5	7517	2395	2193	2193
q6	190	164	130	130
q7	910	742	600	600
q8	9238	1511	1256	1256
q9	5313	5020	4960	4960
q10	6856	2366	1914	1914
q11	489	280	253	253
q12	341	366	221	221
q13	17763	3698	3050	3050
q14	232	236	210	210
q15	525	488	461	461
q16	631	611	596	596
q17	572	898	317	317
q18	7001	6670	6423	6423
q19	1200	975	551	551
q20	312	333	208	208
q21	3014	2161	1975	1975
q22	376	326	308	308
Total cold run time: 102754 ms
Total hot run time: 32511 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5598	5530	5553	5530
q2	258	341	235	235
q3	2297	2659	2293	2293
q4	1439	1784	1332	1332
q5	4359	4813	4678	4678
q6	184	164	134	134
q7	2141	1967	1891	1891
q8	2701	2814	2694	2694
q9	7273	7168	7330	7168
q10	3018	3254	2813	2813
q11	590	514	483	483
q12	711	801	615	615
q13	3417	3943	3200	3200
q14	293	288	275	275
q15	516	462	464	462
q16	650	692	661	661
q17	1266	1721	1268	1268
q18	7717	7547	7244	7244
q19	841	1235	1154	1154
q20	2055	2053	1896	1896
q21	5625	5303	4859	4859
q22	596	591	582	582
Total cold run time: 53545 ms
Total hot run time: 51467 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 184979 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 7f1b3b74f77c9402509a65a343eac3e4ed7cf63c, data reload: false

query1	966	384	381	381
query2	6528	2077	2080	2077
query3	6795	218	215	215
query4	33499	23545	23037	23037
query5	4365	609	454	454
query6	294	193	189	189
query7	4608	489	304	304
query8	292	244	240	240
query9	9649	2702	2699	2699
query10	461	336	255	255
query11	17790	15158	14877	14877
query12	166	107	110	107
query13	1660	521	411	411
query14	10624	6807	6891	6807
query15	247	189	185	185
query16	7947	588	452	452
query17	1640	751	579	579
query18	1962	398	305	305
query19	228	187	164	164
query20	119	117	115	115
query21	218	123	101	101
query22	4259	4377	4350	4350
query23	34132	33132	33125	33125
query24	6431	2207	2215	2207
query25	482	453	388	388
query26	1195	275	152	152
query27	1982	463	329	329
query28	5174	2492	2437	2437
query29	683	532	411	411
query30	232	192	151	151
query31	949	858	804	804
query32	79	64	56	56
query33	518	361	288	288
query34	759	853	519	519
query35	806	824	743	743
query36	998	1035	935	935
query37	119	107	79	79
query38	4247	4291	4075	4075
query39	1487	1389	1406	1389
query40	201	109	106	106
query41	54	52	50	50
query42	138	101	106	101
query43	493	527	486	486
query44	1377	811	822	811
query45	177	171	168	168
query46	848	1056	633	633
query47	1885	1861	1838	1838
query48	377	392	330	330
query49	766	489	384	384
query50	630	654	401	401
query51	4214	4156	4072	4072
query52	110	106	93	93
query53	229	253	183	183
query54	487	497	421	421
query55	79	81	80	80
query56	272	259	242	242
query57	1178	1151	1068	1068
query58	240	238	242	238
query59	3186	3200	2880	2880
query60	276	270	249	249
query61	118	112	119	112
query62	794	715	655	655
query63	214	194	183	183
query64	4297	1000	650	650
query65	3226	3155	3130	3130
query66	1054	462	336	336
query67	16017	15534	15492	15492
query68	4300	821	535	535
query69	468	295	253	253
query70	1215	1146	1148	1146
query71	374	295	267	267
query72	5952	3825	3799	3799
query73	642	751	361	361
query74	10218	9006	8864	8864
query75	3183	3175	2659	2659
query76	3145	1187	790	790
query77	490	352	285	285
query78	10009	10093	9240	9240
query79	2323	820	604	604
query80	672	539	441	441
query81	525	281	236	236
query82	224	154	121	121
query83	185	176	174	174
query84	250	91	76	76
query85	748	350	303	303
query86	401	311	309	309
query87	4388	4369	4405	4369
query88	3960	2184	2216	2184
query89	386	326	289	289
query90	1965	192	190	190
query91	140	134	109	109
query92	69	56	52	52
query93	2088	869	543	543
query94	718	402	307	307
query95	326	268	257	257
query96	482	621	279	279
query97	2779	2813	2700	2700
query98	225	207	200	200
query99	1278	1375	1253	1253
Total cold run time: 281655 ms
Total hot run time: 184979 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.39 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 7f1b3b74f77c9402509a65a343eac3e4ed7cf63c, data reload: false

query1	0.03	0.03	0.06
query2	0.08	0.03	0.03
query3	0.23	0.08	0.07
query4	1.60	0.11	0.11
query5	0.42	0.42	0.40
query6	1.17	0.67	0.66
query7	0.02	0.02	0.02
query8	0.05	0.04	0.03
query9	0.58	0.51	0.52
query10	0.55	0.57	0.56
query11	0.15	0.10	0.10
query12	0.14	0.12	0.11
query13	0.61	0.60	0.60
query14	2.72	2.73	2.84
query15	0.90	0.83	0.83
query16	0.36	0.37	0.39
query17	1.03	1.05	1.00
query18	0.23	0.20	0.21
query19	1.97	1.76	2.02
query20	0.01	0.01	0.01
query21	15.36	0.94	0.57
query22	0.76	0.84	0.60
query23	15.31	1.36	0.59
query24	3.65	1.76	0.69
query25	0.24	0.14	0.13
query26	0.40	0.14	0.14
query27	0.05	0.04	0.04
query28	13.41	0.97	0.43
query29	12.58	3.93	3.31
query30	0.25	0.10	0.06
query31	2.81	0.60	0.38
query32	3.23	0.55	0.47
query33	3.01	3.01	3.01
query34	16.58	5.11	4.52
query35	4.55	4.53	4.52
query36	0.66	0.52	0.49
query37	0.10	0.06	0.06
query38	0.05	0.04	0.03
query39	0.03	0.02	0.02
query40	0.16	0.13	0.12
query41	0.08	0.03	0.03
query42	0.04	0.02	0.02
query43	0.03	0.04	0.03
Total cold run time: 106.19 s
Total hot run time: 30.39 s

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 42.10% (11014/26164)
Line Coverage: 32.38% (92993/287166)
Region Coverage: 31.53% (47677/151223)
Branch Coverage: 27.55% (24123/87576)
Coverage Report: http://coverage.selectdb-in.cc/coverage/7f1b3b74f77c9402509a65a343eac3e4ed7cf63c_7f1b3b74f77c9402509a65a343eac3e4ed7cf63c/report/index.html

HappenLee
HappenLee reviewed Feb 6, 2025
View reviewed changes
be/src/common/config.cpp Outdated Show resolved Hide resolved
@Gabriel39
Copy link
Contributor Author

run buildall

@Gabriel39
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 31640 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 7cbb2c1fe51bdc49d1bbb0b198f5676e737fee9e, data reload: false

------ Round 1 ----------------------------------
q1	17580	5233	5063	5063
q2	2047	290	177	177
q3	10423	1249	752	752
q4	10198	992	531	531
q5	7545	2383	2345	2345
q6	189	166	136	136
q7	911	748	609	609
q8	9313	1328	1074	1074
q9	4960	4628	4767	4628
q10	6814	2312	1906	1906
q11	497	282	264	264
q12	345	350	213	213
q13	17779	3650	3091	3091
q14	235	223	217	217
q15	511	475	454	454
q16	629	614	579	579
q17	579	859	354	354
q18	6661	6303	6265	6265
q19	2921	943	540	540
q20	300	309	189	189
q21	2690	2131	1957	1957
q22	362	332	296	296
Total cold run time: 103489 ms
Total hot run time: 31640 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5266	5155	5074	5074
q2	235	327	230	230
q3	2134	2698	2285	2285
q4	1397	1799	1362	1362
q5	4279	4148	4136	4136
q6	206	166	124	124
q7	1888	1800	1751	1751
q8	2647	2670	2618	2618
q9	7282	7142	7114	7114
q10	3089	3257	2780	2780
q11	580	502	486	486
q12	694	791	639	639
q13	3318	3881	3243	3243
q14	290	295	270	270
q15	503	483	474	474
q16	627	706	638	638
q17	1150	1597	1366	1366
q18	7722	7448	7395	7395
q19	853	851	864	851
q20	1981	2018	1876	1876
q21	5478	5006	4663	4663
q22	610	561	574	561
Total cold run time: 52229 ms
Total hot run time: 49936 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 190643 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 7cbb2c1fe51bdc49d1bbb0b198f5676e737fee9e, data reload: false

query1	1317	958	936	936
query2	6262	1817	1862	1817
query3	10939	4519	4516	4516
query4	54285	24301	22907	22907
query5	4870	576	502	502
query6	323	210	185	185
query7	4899	506	305	305
query8	312	241	220	220
query9	5411	2514	2493	2493
query10	426	325	267	267
query11	15100	15155	14861	14861
query12	160	113	108	108
query13	1040	543	388	388
query14	10303	7224	6831	6831
query15	220	199	185	185
query16	7096	645	473	473
query17	1105	751	615	615
query18	1574	436	326	326
query19	216	207	172	172
query20	137	127	133	127
query21	213	133	110	110
query22	4554	4533	4547	4533
query23	33768	33208	33426	33208
query24	5678	2422	2443	2422
query25	462	456	413	413
query26	719	277	153	153
query27	1917	509	341	341
query28	2860	2418	2440	2418
query29	578	559	436	436
query30	217	188	159	159
query31	898	884	836	836
query32	77	65	69	65
query33	444	367	313	313
query34	792	882	518	518
query35	843	851	770	770
query36	972	1011	912	912
query37	121	103	80	80
query38	4456	4434	4424	4424
query39	1511	1436	1482	1436
query40	211	123	107	107
query41	53	51	49	49
query42	119	105	110	105
query43	517	524	481	481
query44	1372	813	816	813
query45	178	175	173	173
query46	889	1068	671	671
query47	1810	1855	1771	1771
query48	417	433	310	310
query49	700	520	428	428
query50	725	751	425	425
query51	4332	4292	4279	4279
query52	111	110	105	105
query53	247	282	202	202
query54	509	501	419	419
query55	88	80	81	80
query56	279	300	281	281
query57	1173	1165	1135	1135
query58	245	240	248	240
query59	2817	2862	2850	2850
query60	296	291	274	274
query61	121	120	130	120
query62	757	773	704	704
query63	246	196	194	194
query64	1878	1096	710	710
query65	3343	3252	3175	3175
query66	784	393	300	300
query67	15883	15429	15250	15250
query68	6037	785	509	509
query69	539	308	270	270
query70	1231	1106	1143	1106
query71	451	287	278	278
query72	5949	3684	3713	3684
query73	1344	759	347	347
query74	8976	9241	8784	8784
query75	3227	3176	2682	2682
query76	3878	1223	753	753
query77	542	373	295	295
query78	10076	10213	9440	9440
query79	2456	834	592	592
query80	610	525	452	452
query81	497	274	239	239
query82	607	157	121	121
query83	174	181	155	155
query84	290	91	76	76
query85	764	352	306	306
query86	369	293	295	293
query87	4491	4529	4390	4390
query88	3566	2207	2203	2203
query89	422	325	283	283
query90	1748	199	195	195
query91	142	137	107	107
query92	76	61	60	60
query93	1873	1031	579	579
query94	709	412	304	304
query95	345	276	260	260
query96	494	565	265	265
query97	2821	2877	2737	2737
query98	225	204	210	204
query99	1356	1414	1242	1242
Total cold run time: 293460 ms
Total hot run time: 190643 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.05 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 7cbb2c1fe51bdc49d1bbb0b198f5676e737fee9e, data reload: false

query1	0.04	0.04	0.03
query2	0.10	0.03	0.03
query3	0.23	0.07	0.08
query4	1.62	0.10	0.10
query5	0.42	0.41	0.40
query6	1.18	0.66	0.66
query7	0.03	0.01	0.02
query8	0.04	0.03	0.03
query9	0.62	0.51	0.52
query10	0.59	0.58	0.57
query11	0.15	0.11	0.11
query12	0.15	0.11	0.11
query13	0.62	0.61	0.62
query14	2.82	2.82	2.69
query15	0.92	0.84	0.85
query16	0.37	0.36	0.37
query17	1.05	1.05	1.04
query18	0.21	0.19	0.19
query19	1.88	1.81	1.99
query20	0.01	0.01	0.01
query21	15.37	0.89	0.56
query22	0.74	1.19	0.62
query23	14.98	1.40	0.64
query24	6.97	1.70	1.24
query25	0.48	0.10	0.20
query26	0.56	0.17	0.14
query27	0.05	0.05	0.06
query28	10.46	0.90	0.44
query29	12.57	4.05	3.30
query30	0.25	0.10	0.06
query31	2.82	0.60	0.39
query32	3.22	0.54	0.46
query33	2.97	2.99	3.07
query34	15.72	5.15	4.54
query35	4.53	4.54	4.56
query36	0.65	0.52	0.48
query37	0.10	0.06	0.06
query38	0.06	0.04	0.04
query39	0.03	0.03	0.02
query40	0.18	0.14	0.13
query41	0.08	0.02	0.02
query42	0.03	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 105.91 s
Total hot run time: 31.05 s

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 42.08% (11016/26180)
Line Coverage: 32.34% (92987/287497)
Region Coverage: 31.52% (47698/151348)
Branch Coverage: 27.53% (24129/87656)
Coverage Report: http://coverage.selectdb-in.cc/coverage/7cbb2c1fe51bdc49d1bbb0b198f5676e737fee9e_7cbb2c1fe51bdc49d1bbb0b198f5676e737fee9e/report/index.html

@Gabriel39
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 31707 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit f850f21127d1c5c1b98b6783d2ca8e060afe0dfc, data reload: false

------ Round 1 ----------------------------------
q1	17598	5300	5119	5119
q2	2060	311	166	166
q3	10407	1247	752	752
q4	10202	1024	562	562
q5	7537	2360	2427	2360
q6	190	165	133	133
q7	905	757	594	594
q8	9305	1305	1129	1129
q9	4944	4893	4611	4611
q10	6840	2339	1898	1898
q11	471	286	250	250
q12	348	379	228	228
q13	17781	3691	3124	3124
q14	234	225	220	220
q15	516	482	457	457
q16	637	627	572	572
q17	566	865	373	373
q18	6717	6245	6156	6156
q19	1919	961	570	570
q20	317	318	184	184
q21	2869	2161	1948	1948
q22	360	336	301	301
Total cold run time: 102723 ms
Total hot run time: 31707 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5286	5207	5296	5207
q2	243	328	234	234
q3	2205	2684	2283	2283
q4	1485	1888	1396	1396
q5	4280	4169	4184	4169
q6	207	167	128	128
q7	1897	1836	1787	1787
q8	2613	2696	2625	2625
q9	7283	7166	7124	7124
q10	3046	3241	2816	2816
q11	598	519	497	497
q12	695	770	633	633
q13	3479	3979	3288	3288
q14	275	301	275	275
q15	512	454	463	454
q16	634	711	646	646
q17	1164	1632	1353	1353
q18	7692	7258	7254	7254
q19	851	898	942	898
q20	1946	2050	1934	1934
q21	5522	4990	4806	4806
q22	624	573	539	539
Total cold run time: 52537 ms
Total hot run time: 50346 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 191358 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit f850f21127d1c5c1b98b6783d2ca8e060afe0dfc, data reload: false

query1	1322	971	950	950
query2	6229	1896	1922	1896
query3	11071	4645	4592	4592
query4	25875	23419	23462	23419
query5	3712	637	513	513
query6	297	204	186	186
query7	3988	501	295	295
query8	295	260	244	244
query9	8507	2511	2510	2510
query10	468	314	266	266
query11	15490	15179	14994	14994
query12	171	111	105	105
query13	1571	540	415	415
query14	9070	6453	6652	6453
query15	217	198	175	175
query16	7691	646	475	475
query17	1273	807	614	614
query18	2014	420	325	325
query19	215	218	179	179
query20	120	128	123	123
query21	210	125	114	114
query22	4591	4656	4388	4388
query23	34547	33623	33471	33471
query24	7869	2455	2471	2455
query25	539	482	407	407
query26	1173	286	160	160
query27	2522	488	337	337
query28	4655	2433	2398	2398
query29	694	574	441	441
query30	223	191	158	158
query31	929	885	784	784
query32	77	72	65	65
query33	546	356	319	319
query34	784	888	523	523
query35	839	842	788	788
query36	997	1026	905	905
query37	124	98	98	98
query38	4397	4458	4435	4435
query39	1497	1435	1458	1435
query40	206	116	110	110
query41	52	54	49	49
query42	121	113	109	109
query43	544	515	488	488
query44	1272	800	812	800
query45	196	174	164	164
query46	911	1070	670	670
query47	1833	1861	1781	1781
query48	384	433	317	317
query49	771	537	426	426
query50	710	754	434	434
query51	4432	4289	4243	4243
query52	108	109	99	99
query53	252	275	194	194
query54	507	516	428	428
query55	87	89	79	79
query56	264	291	268	268
query57	1170	1210	1105	1105
query58	245	250	240	240
query59	2908	2859	2865	2859
query60	304	280	269	269
query61	121	159	136	136
query62	798	755	683	683
query63	236	197	196	196
query64	4158	1034	697	697
query65	3345	3238	3251	3238
query66	912	406	312	312
query67	16220	15644	15483	15483
query68	8384	783	524	524
query69	467	298	263	263
query70	1206	1171	1116	1116
query71	436	303	292	292
query72	5117	3524	3811	3524
query73	737	720	344	344
query74	8975	9197	8818	8818
query75	3248	3195	2704	2704
query76	3401	1180	761	761
query77	563	381	285	285
query78	10055	9957	9272	9272
query79	2753	860	600	600
query80	634	546	445	445
query81	494	285	243	243
query82	470	150	123	123
query83	172	170	150	150
query84	235	99	69	69
query85	789	350	306	306
query86	366	332	261	261
query87	4431	4741	4432	4432
query88	3866	2178	2165	2165
query89	407	336	289	289
query90	1959	194	192	192
query91	135	137	107	107
query92	75	61	56	56
query93	2479	1028	598	598
query94	654	390	307	307
query95	358	270	269	269
query96	483	560	270	270
query97	2791	2848	2748	2748
query98	233	207	203	203
query99	1330	1450	1335	1335
Total cold run time: 277579 ms
Total hot run time: 191358 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.12 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit f850f21127d1c5c1b98b6783d2ca8e060afe0dfc, data reload: false

query1	0.04	0.04	0.04
query2	0.07	0.04	0.03
query3	0.24	0.06	0.07
query4	1.64	0.10	0.11
query5	0.40	0.41	0.41
query6	1.15	0.66	0.66
query7	0.02	0.02	0.02
query8	0.04	0.03	0.03
query9	0.58	0.52	0.51
query10	0.57	0.59	0.55
query11	0.15	0.10	0.10
query12	0.14	0.11	0.11
query13	0.62	0.60	0.62
query14	2.73	2.85	2.69
query15	0.92	0.85	0.87
query16	0.37	0.39	0.39
query17	1.02	1.00	1.04
query18	0.22	0.19	0.19
query19	1.92	1.78	1.98
query20	0.02	0.01	0.02
query21	15.35	0.94	0.56
query22	0.76	1.15	0.70
query23	14.92	1.34	0.65
query24	6.87	1.36	1.26
query25	0.51	0.23	0.06
query26	0.50	0.17	0.14
query27	0.06	0.05	0.05
query28	9.61	0.91	0.44
query29	12.54	3.95	3.31
query30	0.24	0.08	0.06
query31	2.83	0.59	0.37
query32	3.24	0.55	0.46
query33	2.97	3.03	3.03
query34	15.78	5.13	4.52
query35	4.54	4.57	4.58
query36	0.66	0.49	0.47
query37	0.10	0.07	0.07
query38	0.05	0.04	0.04
query39	0.03	0.02	0.02
query40	0.17	0.14	0.14
query41	0.08	0.03	0.02
query42	0.04	0.02	0.03
query43	0.03	0.04	0.03
Total cold run time: 104.74 s
Total hot run time: 31.12 s

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 42.07% (11018/26187)
Line Coverage: 32.34% (92993/287549)
Region Coverage: 31.52% (47706/151362)
Branch Coverage: 27.52% (24129/87664)
Coverage Report: http://coverage.selectdb-in.cc/coverage/f850f21127d1c5c1b98b6783d2ca8e060afe0dfc_f850f21127d1c5c1b98b6783d2ca8e060afe0dfc/report/index.html

@Gabriel39
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 31542 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 9e5afe84b9e182494e12aab3f0d6f79dd52e8148, data reload: false

------ Round 1 ----------------------------------
q1	17674	5282	5076	5076
q2	2054	311	169	169
q3	10385	1334	700	700
q4	10220	1026	537	537
q5	7558	2429	2380	2380
q6	192	165	135	135
q7	916	736	610	610
q8	9286	1321	1100	1100
q9	5695	4688	4820	4688
q10	6844	2328	1920	1920
q11	472	276	249	249
q12	348	367	219	219
q13	17767	3708	3026	3026
q14	219	230	210	210
q15	509	480	462	462
q16	624	612	590	590
q17	575	884	346	346
q18	7210	6305	6162	6162
q19	1219	962	552	552
q20	321	324	206	206
q21	2838	2165	1902	1902
q22	365	321	303	303
Total cold run time: 103291 ms
Total hot run time: 31542 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5106	5133	5113	5113
q2	233	328	240	240
q3	2151	2682	2271	2271
q4	1395	1818	1361	1361
q5	4244	4143	4188	4143
q6	205	163	126	126
q7	1868	1814	1810	1810
q8	2668	2687	2544	2544
q9	7317	7220	7201	7201
q10	3029	3156	2773	2773
q11	589	509	486	486
q12	710	767	606	606
q13	3527	3875	3240	3240
q14	301	320	265	265
q15	496	471	452	452
q16	653	671	642	642
q17	1144	1601	1329	1329
q18	7636	7469	7364	7364
q19	805	827	866	827
q20	1998	1981	1852	1852
q21	5422	4965	4700	4700
q22	640	596	587	587
Total cold run time: 52137 ms
Total hot run time: 49932 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 183810 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 9e5afe84b9e182494e12aab3f0d6f79dd52e8148, data reload: false

query1	969	388	397	388
query2	6559	1850	1838	1838
query3	6806	219	214	214
query4	26527	23613	22976	22976
query5	4308	627	463	463
query6	279	197	182	182
query7	4610	513	309	309
query8	306	239	251	239
query9	8637	2502	2499	2499
query10	443	331	261	261
query11	15740	15295	15029	15029
query12	162	110	109	109
query13	1654	528	398	398
query14	9200	6608	6851	6608
query15	215	197	189	189
query16	7783	619	456	456
query17	1207	718	566	566
query18	1947	402	305	305
query19	198	204	162	162
query20	123	117	118	117
query21	216	122	108	108
query22	4240	4403	4303	4303
query23	34000	33475	32987	32987
query24	7740	2351	2416	2351
query25	471	452	390	390
query26	1069	240	153	153
query27	2134	495	329	329
query28	3906	2387	2359	2359
query29	564	504	398	398
query30	239	186	162	162
query31	949	862	817	817
query32	78	63	65	63
query33	548	383	304	304
query34	797	850	502	502
query35	797	819	730	730
query36	956	997	878	878
query37	125	94	88	88
query38	4232	4134	4069	4069
query39	1493	1420	1409	1409
query40	203	127	103	103
query41	53	54	50	50
query42	127	103	103	103
query43	502	502	481	481
query44	1263	771	765	765
query45	175	172	161	161
query46	864	1040	637	637
query47	1806	1847	1741	1741
query48	371	407	301	301
query49	760	510	417	417
query50	715	739	429	429
query51	4177	4235	4161	4161
query52	113	103	99	99
query53	225	254	178	178
query54	492	493	397	397
query55	81	77	79	77
query56	273	274	257	257
query57	1103	1158	1067	1067
query58	245	251	237	237
query59	2566	2763	2796	2763
query60	280	265	270	265
query61	124	112	133	112
query62	804	717	671	671
query63	224	182	184	182
query64	4338	1002	657	657
query65	3252	3129	3122	3122
query66	1076	415	299	299
query67	15964	15551	15463	15463
query68	8014	756	503	503
query69	484	299	263	263
query70	1206	1146	1091	1091
query71	390	289	275	275
query72	5692	3513	3824	3513
query73	739	729	350	350
query74	8977	9284	8926	8926
query75	3171	3189	2696	2696
query76	3261	1171	776	776
query77	504	363	287	287
query78	10046	10140	9336	9336
query79	2671	800	607	607
query80	639	522	446	446
query81	517	270	233	233
query82	488	151	118	118
query83	176	170	151	151
query84	260	100	73	73
query85	801	348	312	312
query86	377	295	270	270
query87	4488	4440	4335	4335
query88	3957	2174	2167	2167
query89	430	317	291	291
query90	1901	195	189	189
query91	135	135	112	112
query92	70	69	55	55
query93	2316	1004	574	574
query94	676	399	299	299
query95	361	274	255	255
query96	479	552	266	266
query97	2840	2845	2687	2687
query98	224	205	205	205
query99	1351	1416	1246	1246
Total cold run time: 272044 ms
Total hot run time: 183810 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.82 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 9e5afe84b9e182494e12aab3f0d6f79dd52e8148, data reload: false

query1	0.04	0.04	0.03
query2	0.07	0.03	0.03
query3	0.24	0.06	0.07
query4	1.62	0.10	0.10
query5	0.42	0.41	0.39
query6	1.16	0.67	0.65
query7	0.02	0.02	0.02
query8	0.04	0.04	0.03
query9	0.60	0.52	0.52
query10	0.56	0.58	0.57
query11	0.15	0.10	0.10
query12	0.15	0.11	0.10
query13	0.62	0.60	0.59
query14	2.68	2.70	2.81
query15	0.93	0.86	0.86
query16	0.39	0.39	0.37
query17	1.04	1.01	1.02
query18	0.22	0.19	0.19
query19	1.97	1.80	2.01
query20	0.01	0.02	0.01
query21	15.37	0.91	0.55
query22	0.76	1.15	0.63
query23	15.04	1.38	0.65
query24	7.08	1.57	1.03
query25	0.50	0.20	0.13
query26	0.64	0.16	0.13
query27	0.05	0.04	0.05
query28	9.62	0.89	0.42
query29	12.59	4.04	3.38
query30	0.25	0.09	0.06
query31	2.84	0.61	0.39
query32	3.22	0.55	0.46
query33	3.00	3.00	3.02
query34	15.76	5.12	4.52
query35	4.56	4.50	4.52
query36	0.68	0.50	0.48
query37	0.10	0.07	0.06
query38	0.05	0.03	0.03
query39	0.04	0.03	0.02
query40	0.17	0.14	0.14
query41	0.08	0.03	0.02
query42	0.04	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 105.41 s
Total hot run time: 30.82 s

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 42.08% (11024/26199)
Line Coverage: 32.34% (93021/287676)
Region Coverage: 31.51% (47713/151405)
Branch Coverage: 27.53% (24137/87682)
Coverage Report: http://coverage.selectdb-in.cc/coverage/9e5afe84b9e182494e12aab3f0d6f79dd52e8148_9e5afe84b9e182494e12aab3f0d6f79dd52e8148/report/index.html

HappenLee
HappenLee approved these changes Feb 7, 2025
View reviewed changes
Copy link
Contributor

@HappenLee HappenLee left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Feb 7, 2025
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Feb 7, 2025

PR approved by at least one committer and no changes requested.

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Feb 7, 2025

PR approved by anyone and no changes requested.

@Gabriel39 Gabriel39 merged commit f3e36c2 into apache:master Feb 7, 2025
23 of 26 checks passed
Gabriel39 added a commit to Gabriel39/incubator-doris that referenced this pull request Feb 7, 2025
@Gabriel39
[Improvement](tablet) Prefetch tablets before opening (apache#47501)
306d613 
For a scan operator in a query, FE will assign a tablet id to a specific
scan operator and then it will be search by storage engine during
execution. But this searching process will be done after runtime filter
reached. However, if we wait for a runtime filter for a long time and
compaction / balance tasks is done at the same time, a tablet / rowset
will be lost and we got an error.
@Gabriel39 Gabriel39 mentioned this pull request Feb 7, 2025
Merged
16 tasks
Gabriel39 added a commit that referenced this pull request Feb 8, 2025
@Gabriel39
[branch-3.0](pick) pick #47501 #47535 (#47602)
9305725
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@BiteTheDDDDt BiteTheDDDDt BiteTheDDDDt approved these changes

@HappenLee HappenLee HappenLee approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/3.0.4-merged reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@Gabriel39 @Thearas @doris-robot @BiteTheDDDDt @HappenLee