Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[feature](inverted index) Add profile statistics for each condition in inverted index filters #47504

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open

[feature](inverted index) Add profile statistics for each condition in inverted index filters #47504

wants to merge 1 commit into from
+244 −2

Conversation

zzzxl1993
Copy link
Contributor

@zzzxl1993 zzzxl1993 commented Feb 5, 2025 •
edited
Loading

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

IndexFilter:
- InvertedIndexFilter: [(clientip, 3, 410.930us), (request, 863, 466.821us), (request, 350, 165.685us)]

This output shows the performance of the index filter across different columns. Each entry includes the following information:

Column Name: The name of the column on which the index filtering operation was applied.
Filtered Rows: The number of rows filtered in the respective column, indicating the extent of the filtering operation.
Filter Time: The time taken to perform the filtering operation on the column, measured in microseconds (μs), reflecting the efficiency of the filtering process.

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@Thearas
Copy link
Contributor

Thearas commented Feb 5, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@zzzxl1993
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 32210 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b35d7498119286144b585d6dc786458fc42d6a52, data reload: false

------ Round 1 ----------------------------------
q1	17584	5578	5403	5403
q2	2052	308	170	170
q3	10482	1209	755	755
q4	10203	987	527	527
q5	7532	2428	2144	2144
q6	195	170	136	136
q7	939	743	599	599
q8	9242	1401	1152	1152
q9	5193	5217	4855	4855
q10	6840	2341	1936	1936
q11	472	273	253	253
q12	341	362	218	218
q13	17752	3671	3110	3110
q14	233	222	202	202
q15	520	473	466	466
q16	613	618	585	585
q17	572	876	332	332
q18	7242	6363	6371	6363
q19	2757	969	540	540
q20	305	313	188	188
q21	2892	2171	1971	1971
q22	370	327	305	305
Total cold run time: 104331 ms
Total hot run time: 32210 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5653	5512	5501	5501
q2	242	330	237	237
q3	2288	2648	2322	2322
q4	1452	1816	1369	1369
q5	4301	4747	4666	4666
q6	172	161	131	131
q7	2081	2010	1859	1859
q8	2593	2897	2711	2711
q9	7268	7255	7245	7245
q10	3051	3305	2767	2767
q11	570	503	478	478
q12	646	756	571	571
q13	3520	3981	3271	3271
q14	279	308	273	273
q15	522	483	462	462
q16	650	674	664	664
q17	1233	1753	1270	1270
q18	7777	7554	7277	7277
q19	816	1163	1117	1117
q20	1973	2020	1910	1910
q21	5812	5431	5087	5087
q22	598	623	540	540
Total cold run time: 53497 ms
Total hot run time: 51728 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 193009 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit b35d7498119286144b585d6dc786458fc42d6a52, data reload: false

query1	1342	959	910	910
query2	6317	2089	1998	1998
query3	11111	4591	4550	4550
query4	61359	29198	23508	23508
query5	5415	576	453	453
query6	447	210	185	185
query7	5537	512	301	301
query8	339	245	241	241
query9	8195	2620	2598	2598
query10	466	312	255	255
query11	17343	15407	15377	15377
query12	175	115	114	114
query13	1425	559	427	427
query14	10678	7828	7855	7828
query15	215	205	183	183
query16	7230	664	484	484
query17	1162	740	582	582
query18	1832	409	319	319
query19	214	191	162	162
query20	115	114	110	110
query21	220	128	109	109
query22	4504	4476	4769	4476
query23	34038	33501	34145	33501
query24	5878	2294	2395	2294
query25	462	447	412	412
query26	735	237	159	159
query27	2034	508	328	328
query28	3938	2474	2446	2446
query29	543	545	422	422
query30	207	188	152	152
query31	954	887	819	819
query32	71	62	54	54
query33	477	389	299	299
query34	770	889	562	562
query35	815	844	745	745
query36	1003	1070	968	968
query37	127	101	78	78
query38	4323	4312	4218	4218
query39	1520	1435	1419	1419
query40	201	114	99	99
query41	51	49	53	49
query42	121	107	106	106
query43	515	537	498	498
query44	1375	852	852	852
query45	182	186	171	171
query46	887	1056	664	664
query47	1945	1927	1862	1862
query48	384	396	320	320
query49	738	493	414	414
query50	680	688	396	396
query51	4275	4318	4229	4229
query52	105	104	94	94
query53	230	260	185	185
query54	502	516	420	420
query55	82	83	79	79
query56	257	269	251	251
query57	1185	1261	1155	1155
query58	258	239	249	239
query59	3115	3203	3128	3128
query60	293	290	266	266
query61	126	114	120	114
query62	757	701	663	663
query63	222	191	180	180
query64	1822	1028	655	655
query65	3241	3121	3144	3121
query66	749	401	301	301
query67	16185	15717	15499	15499
query68	4999	874	522	522
query69	482	301	254	254
query70	1194	1149	1137	1137
query71	406	287	259	259
query72	6044	3927	3809	3809
query73	788	757	370	370
query74	9805	8904	8607	8607
query75	3225	3163	2651	2651
query76	3799	1150	772	772
query77	482	367	266	266
query78	10039	9892	9421	9421
query79	2789	853	596	596
query80	1697	518	434	434
query81	564	277	236	236
query82	350	147	120	120
query83	270	174	155	155
query84	286	97	69	69
query85	755	355	306	306
query86	454	321	306	306
query87	4403	4397	4301	4301
query88	3910	2205	2206	2205
query89	389	322	298	298
query90	1594	184	183	183
query91	175	130	103	103
query92	66	58	53	53
query93	2926	866	518	518
query94	734	414	305	305
query95	323	265	252	252
query96	493	597	288	288
query97	2838	2882	2709	2709
query98	235	205	196	196
query99	1316	1364	1252	1252
Total cold run time: 311849 ms
Total hot run time: 193009 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.25 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit b35d7498119286144b585d6dc786458fc42d6a52, data reload: false

query1	0.03	0.03	0.03
query2	0.07	0.03	0.03
query3	0.23	0.07	0.07
query4	1.62	0.11	0.10
query5	0.42	0.42	0.40
query6	1.17	0.66	0.65
query7	0.02	0.02	0.02
query8	0.04	0.03	0.03
query9	0.58	0.50	0.50
query10	0.55	0.56	0.56
query11	0.15	0.11	0.10
query12	0.15	0.12	0.12
query13	0.62	0.59	0.60
query14	2.75	2.73	2.79
query15	0.89	0.83	0.83
query16	0.39	0.39	0.38
query17	1.05	1.06	1.02
query18	0.22	0.20	0.20
query19	1.93	1.89	2.02
query20	0.01	0.01	0.01
query21	15.41	0.94	0.59
query22	0.75	0.81	0.70
query23	15.26	1.36	0.51
query24	2.69	1.33	0.49
query25	0.16	0.22	0.04
query26	0.40	0.16	0.13
query27	0.07	0.06	0.04
query28	12.97	1.03	0.43
query29	12.57	4.01	3.31
query30	0.25	0.08	0.06
query31	2.81	0.60	0.38
query32	3.23	0.55	0.46
query33	2.94	3.04	3.04
query34	16.58	5.26	4.51
query35	4.56	4.51	4.51
query36	0.65	0.49	0.48
query37	0.10	0.06	0.05
query38	0.05	0.04	0.03
query39	0.03	0.02	0.03
query40	0.17	0.14	0.13
query41	0.08	0.03	0.02
query42	0.04	0.03	0.03
query43	0.03	0.03	0.03
Total cold run time: 104.69 s
Total hot run time: 30.25 s

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 42.10% (11006/26145)
Line Coverage: 32.37% (92920/287072)
Region Coverage: 31.51% (47634/151158)
Branch Coverage: 27.54% (24106/87542)
Coverage Report: http://coverage.selectdb-in.cc/coverage/b35d7498119286144b585d6dc786458fc42d6a52_b35d7498119286144b585d6dc786458fc42d6a52/report/index.html

@zzzxl1993
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 32686 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b35d7498119286144b585d6dc786458fc42d6a52, data reload: false

------ Round 1 ----------------------------------
q1	17635	5532	5430	5430
q2	2064	335	191	191
q3	10454	1327	746	746
q4	10229	977	535	535
q5	7681	2479	2239	2239
q6	199	168	135	135
q7	932	798	605	605
q8	9247	1433	1266	1266
q9	5279	4949	4912	4912
q10	6818	2349	1875	1875
q11	463	280	257	257
q12	352	358	223	223
q13	17776	3729	3083	3083
q14	232	230	223	223
q15	517	465	479	465
q16	630	604	589	589
q17	596	891	335	335
q18	7089	6534	6512	6512
q19	1861	966	562	562
q20	328	342	183	183
q21	2988	2228	2019	2019
q22	361	330	301	301
Total cold run time: 103731 ms
Total hot run time: 32686 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5631	5608	5548	5548
q2	244	323	231	231
q3	2307	2639	2310	2310
q4	1448	1875	1359	1359
q5	4324	4786	4773	4773
q6	181	168	133	133
q7	2177	2023	1834	1834
q8	2641	2841	2722	2722
q9	7388	7271	7307	7271
q10	2976	3289	2817	2817
q11	580	524	488	488
q12	674	739	616	616
q13	3517	3913	3328	3328
q14	277	297	278	278
q15	505	486	469	469
q16	660	670	643	643
q17	1249	1740	1245	1245
q18	7801	7507	7346	7346
q19	854	1150	1094	1094
q20	2049	2045	1957	1957
q21	5690	5226	4963	4963
q22	567	614	546	546
Total cold run time: 53740 ms
Total hot run time: 51971 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 186506 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit b35d7498119286144b585d6dc786458fc42d6a52, data reload: false

query1	978	382	361	361
query2	6525	2081	2070	2070
query3	6795	226	216	216
query4	33532	23479	23639	23479
query5	4399	637	465	465
query6	294	209	198	198
query7	4607	491	310	310
query8	300	246	229	229
query9	9660	2663	2639	2639
query10	467	329	266	266
query11	17905	15289	15366	15289
query12	175	108	104	104
query13	1679	526	385	385
query14	11434	8044	7263	7263
query15	212	199	197	197
query16	7113	615	460	460
query17	1584	729	546	546
query18	1362	370	299	299
query19	238	181	161	161
query20	117	109	110	109
query21	215	121	99	99
query22	4177	4310	4193	4193
query23	33805	32859	33407	32859
query24	6266	2325	2291	2291
query25	480	450	385	385
query26	789	274	157	157
query27	1972	468	330	330
query28	5295	2442	2423	2423
query29	530	565	468	468
query30	233	186	163	163
query31	996	874	785	785
query32	72	67	63	63
query33	514	354	307	307
query34	755	875	521	521
query35	861	829	737	737
query36	992	1044	929	929
query37	118	96	78	78
query38	4136	4090	4027	4027
query39	1459	1414	1435	1414
query40	211	115	102	102
query41	52	55	52	52
query42	128	104	103	103
query43	521	559	499	499
query44	1431	807	806	806
query45	180	169	165	165
query46	867	1044	646	646
query47	1821	1814	1766	1766
query48	386	406	316	316
query49	743	480	401	401
query50	669	658	395	395
query51	4176	4199	4119	4119
query52	107	102	91	91
query53	238	265	188	188
query54	487	497	419	419
query55	85	79	78	78
query56	280	282	247	247
query57	1149	1159	1080	1080
query58	256	234	240	234
query59	3047	3229	3167	3167
query60	280	261	254	254
query61	122	117	121	117
query62	805	716	660	660
query63	224	199	194	194
query64	3266	1016	657	657
query65	3263	3154	3168	3154
query66	878	418	307	307
query67	15648	15589	15557	15557
query68	2461	835	551	551
query69	423	298	265	265
query70	1198	1172	1126	1126
query71	322	295	259	259
query72	5068	4037	3881	3881
query73	630	756	361	361
query74	9834	9033	8909	8909
query75	3160	3207	2661	2661
query76	2145	1198	797	797
query77	353	375	305	305
query78	10053	10360	9273	9273
query79	998	915	596	596
query80	710	551	455	455
query81	477	278	236	236
query82	1141	155	126	126
query83	288	183	157	157
query84	239	88	78	78
query85	717	383	302	302
query86	340	307	303	303
query87	4400	4432	4382	4382
query88	3468	2194	2149	2149
query89	400	335	296	296
query90	1631	194	197	194
query91	137	145	112	112
query92	60	59	55	55
query93	991	909	536	536
query94	479	409	304	304
query95	337	276	268	268
query96	511	605	280	280
query97	2826	2870	2782	2782
query98	211	209	199	199
query99	1268	1462	1259	1259
Total cold run time: 271637 ms
Total hot run time: 186506 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.65 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit b35d7498119286144b585d6dc786458fc42d6a52, data reload: false

query1	0.03	0.03	0.03
query2	0.08	0.03	0.04
query3	0.24	0.06	0.07
query4	1.61	0.10	0.11
query5	0.41	0.40	0.41
query6	1.13	0.65	0.66
query7	0.03	0.02	0.01
query8	0.04	0.03	0.03
query9	0.59	0.50	0.50
query10	0.56	0.56	0.56
query11	0.14	0.11	0.10
query12	0.14	0.11	0.11
query13	0.61	0.60	0.60
query14	2.77	2.86	2.93
query15	0.89	0.82	0.84
query16	0.40	0.38	0.36
query17	1.05	0.98	1.00
query18	0.23	0.21	0.22
query19	1.96	1.89	2.01
query20	0.02	0.01	0.02
query21	15.36	0.90	0.58
query22	0.76	0.74	0.76
query23	15.24	1.48	0.51
query24	3.00	0.91	1.48
query25	0.14	0.21	0.06
query26	0.22	0.16	0.14
query27	0.06	0.05	0.04
query28	13.98	0.98	0.44
query29	12.59	4.01	3.30
query30	0.25	0.09	0.06
query31	2.84	0.57	0.39
query32	3.25	0.56	0.46
query33	3.02	3.02	3.02
query34	16.57	5.11	4.43
query35	4.53	4.47	4.51
query36	0.63	0.49	0.49
query37	0.10	0.06	0.06
query38	0.04	0.03	0.03
query39	0.03	0.02	0.03
query40	0.18	0.13	0.12
query41	0.08	0.03	0.02
query42	0.04	0.03	0.02
query43	0.03	0.04	0.03
Total cold run time: 105.87 s
Total hot run time: 30.65 s

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 42.11% (11017/26164)
Line Coverage: 32.38% (92993/287187)
Region Coverage: 31.54% (47703/151223)
Branch Coverage: 27.55% (24129/87578)
Coverage Report: http://coverage.selectdb-in.cc/coverage/b35d7498119286144b585d6dc786458fc42d6a52_b35d7498119286144b585d6dc786458fc42d6a52/report/index.html

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@zzzxl1993 @Thearas @doris-robot