Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Refactor](UDF) Refactor the java udf code to reduce the unless code #47505

Merged
merged 1 commit into from
Feb 6, 2025

Conversation

HappenLee
Copy link
Contributor

@HappenLee HappenLee commented Feb 5, 2025

What problem does this PR solve?

Refactor the java udf code to reduce the unless code

  1. JavaUdfDataTypeSet to JavaUdfDataTypeMap speed up is support search
  2. Rewrite the JavaUdfDataType toString()
  3. remove the unless getKeyType and setKeyType
  4. remove the unless child method:getInputConverters/getOutputConverter

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen
Copy link
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@HappenLee
Copy link
Contributor Author

run buildall

@HappenLee
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 32172 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ab435262d95d286f0066b778942626a108008594, data reload: false

------ Round 1 ----------------------------------
q1	17572	5444	5322	5322
q2	2051	292	162	162
q3	10466	1226	751	751
q4	10468	980	534	534
q5	9575	2432	2203	2203
q6	199	167	135	135
q7	890	754	601	601
q8	9241	1355	1170	1170
q9	5159	4938	4841	4841
q10	6836	2342	1902	1902
q11	465	274	253	253
q12	340	365	216	216
q13	17766	3671	3112	3112
q14	228	234	221	221
q15	509	475	461	461
q16	629	611	599	599
q17	558	873	323	323
q18	6834	6472	6381	6381
q19	3476	951	540	540
q20	303	312	183	183
q21	2754	2149	1958	1958
q22	366	328	304	304
Total cold run time: 106685 ms
Total hot run time: 32172 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5617	5477	5451	5451
q2	237	326	228	228
q3	2277	2656	2297	2297
q4	1418	1802	1358	1358
q5	4249	4725	4778	4725
q6	174	168	129	129
q7	2062	1995	1826	1826
q8	2587	2847	2714	2714
q9	7328	7291	7315	7291
q10	3026	3322	2687	2687
q11	565	509	524	509
q12	688	806	633	633
q13	3531	3868	3291	3291
q14	295	306	274	274
q15	505	489	463	463
q16	653	684	677	677
q17	1220	1749	1263	1263
q18	7640	7380	7284	7284
q19	804	1145	1067	1067
q20	1985	2061	1884	1884
q21	5807	4990	4974	4974
q22	599	578	574	574
Total cold run time: 53267 ms
Total hot run time: 51599 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 190785 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ab435262d95d286f0066b778942626a108008594, data reload: false

query1	1294	942	957	942
query2	6464	2026	2021	2021
query3	11107	4636	4665	4636
query4	32824	23940	22990	22990
query5	4170	628	462	462
query6	289	210	196	196
query7	3984	490	315	315
query8	292	234	225	225
query9	9280	2647	2636	2636
query10	468	340	259	259
query11	17743	15207	14975	14975
query12	175	107	106	106
query13	1564	550	414	414
query14	9941	6349	7069	6349
query15	259	213	188	188
query16	7812	659	474	474
query17	1564	734	591	591
query18	2157	412	319	319
query19	212	186	159	159
query20	122	111	131	111
query21	207	121	105	105
query22	4428	4496	4474	4474
query23	34438	33429	33083	33083
query24	6562	2348	2320	2320
query25	514	474	376	376
query26	820	280	164	164
query27	2171	472	339	339
query28	5628	2458	2463	2458
query29	614	572	455	455
query30	217	193	153	153
query31	978	873	810	810
query32	79	59	58	58
query33	481	367	305	305
query34	765	882	521	521
query35	807	891	775	775
query36	997	1061	967	967
query37	123	107	80	80
query38	4357	4298	4260	4260
query39	1492	1458	1449	1449
query40	210	115	101	101
query41	51	55	48	48
query42	128	101	103	101
query43	503	532	500	500
query44	1364	827	832	827
query45	182	183	169	169
query46	884	1075	655	655
query47	1880	1930	1886	1886
query48	381	410	344	344
query49	707	528	427	427
query50	675	714	414	414
query51	4388	4397	4199	4199
query52	102	110	93	93
query53	235	261	201	201
query54	488	517	437	437
query55	83	84	79	79
query56	259	283	244	244
query57	1196	1173	1142	1142
query58	236	235	234	234
query59	3039	3199	2987	2987
query60	298	277	259	259
query61	125	113	116	113
query62	802	734	697	697
query63	231	204	194	194
query64	3432	1015	644	644
query65	3322	3309	3250	3250
query66	1053	406	310	310
query67	16174	15462	15302	15302
query68	1980	845	543	543
query69	406	295	258	258
query70	1200	1149	1051	1051
query71	305	287	290	287
query72	6170	3845	3859	3845
query73	626	747	360	360
query74	9980	9099	9320	9099
query75	3174	3160	2664	2664
query76	2027	1172	776	776
query77	341	359	271	271
query78	10027	10305	9278	9278
query79	996	930	614	614
query80	1361	541	449	449
query81	548	274	244	244
query82	215	150	124	124
query83	277	170	153	153
query84	232	91	72	72
query85	1025	356	296	296
query86	393	324	283	283
query87	4613	4464	4326	4326
query88	3455	2195	2142	2142
query89	395	344	304	304
query90	1576	197	194	194
query91	140	145	108	108
query92	59	58	56	56
query93	965	862	557	557
query94	652	416	278	278
query95	340	274	265	265
query96	487	669	292	292
query97	2865	2853	2746	2746
query98	235	199	201	199
query99	1274	1391	1274	1274
Total cold run time: 278166 ms
Total hot run time: 190785 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.68 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit ab435262d95d286f0066b778942626a108008594, data reload: false

query1	0.03	0.03	0.03
query2	0.08	0.05	0.03
query3	0.24	0.07	0.06
query4	1.62	0.11	0.10
query5	0.40	0.41	0.41
query6	1.16	0.66	0.65
query7	0.02	0.02	0.02
query8	0.04	0.04	0.03
query9	0.56	0.50	0.51
query10	0.55	0.55	0.55
query11	0.14	0.10	0.10
query12	0.14	0.10	0.11
query13	0.61	0.59	0.60
query14	2.71	2.84	2.77
query15	0.90	0.82	0.83
query16	0.40	0.39	0.38
query17	1.05	1.00	1.05
query18	0.22	0.21	0.21
query19	1.94	1.77	1.98
query20	0.01	0.01	0.01
query21	15.36	0.96	0.59
query22	0.75	0.85	0.73
query23	15.17	1.40	0.51
query24	2.62	1.72	0.83
query25	0.15	0.17	0.17
query26	0.32	0.15	0.14
query27	0.09	0.05	0.07
query28	14.33	1.05	0.43
query29	12.58	3.90	3.27
query30	0.26	0.10	0.06
query31	2.81	0.60	0.38
query32	3.23	0.55	0.48
query33	3.08	3.07	3.14
query34	16.64	5.17	4.54
query35	4.57	4.51	4.56
query36	0.67	0.50	0.47
query37	0.09	0.06	0.06
query38	0.04	0.04	0.03
query39	0.04	0.02	0.02
query40	0.16	0.13	0.14
query41	0.09	0.03	0.03
query42	0.04	0.03	0.02
query43	0.03	0.03	0.03
Total cold run time: 105.94 s
Total hot run time: 30.68 s

@HappenLee HappenLee force-pushed the poc branch 2 times, most recently from cefc3d2 to c13c352 Compare February 5, 2025 10:15
@HappenLee
Copy link
Contributor Author

run buildall

@HappenLee
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 32570 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit dcf927d6cc8bce0a090127c9feadec36019be7e6, data reload: false

------ Round 1 ----------------------------------
q1	17577	5539	5853	5539
q2	2064	309	173	173
q3	10424	1261	745	745
q4	10201	967	511	511
q5	7532	2443	2160	2160
q6	195	169	135	135
q7	905	781	609	609
q8	9240	1394	1163	1163
q9	5522	4989	4932	4932
q10	6856	2326	1886	1886
q11	474	281	264	264
q12	353	373	225	225
q13	17762	3705	3133	3133
q14	239	244	227	227
q15	519	485	466	466
q16	652	621	585	585
q17	585	879	334	334
q18	6948	6572	6408	6408
q19	1224	958	539	539
q20	330	333	218	218
q21	2893	2237	2004	2004
q22	375	332	314	314
Total cold run time: 102870 ms
Total hot run time: 32570 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5570	5527	5519	5519
q2	239	335	236	236
q3	2255	2648	2330	2330
q4	1417	1844	1404	1404
q5	4310	4746	4619	4619
q6	165	156	129	129
q7	2121	1965	1821	1821
q8	2667	2811	2673	2673
q9	7232	7197	7204	7197
q10	2962	3266	2756	2756
q11	585	525	507	507
q12	669	782	610	610
q13	3548	3952	3302	3302
q14	286	310	266	266
q15	518	480	488	480
q16	673	691	651	651
q17	1217	1745	1243	1243
q18	7557	7368	7391	7368
q19	773	898	1124	898
q20	2015	2023	1909	1909
q21	5705	5427	4887	4887
q22	598	582	578	578
Total cold run time: 53082 ms
Total hot run time: 51383 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 184829 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit dcf927d6cc8bce0a090127c9feadec36019be7e6, data reload: false

query1	954	378	400	378
query2	6528	2067	2033	2033
query3	6795	215	221	215
query4	33148	23313	23049	23049
query5	4421	630	482	482
query6	295	217	191	191
query7	4600	501	307	307
query8	285	234	234	234
query9	9539	2687	2673	2673
query10	457	306	250	250
query11	17923	15268	14900	14900
query12	156	111	113	111
query13	1652	526	408	408
query14	9997	6557	6560	6557
query15	218	217	188	188
query16	7956	643	470	470
query17	1600	752	589	589
query18	2077	419	311	311
query19	224	184	164	164
query20	120	112	117	112
query21	210	127	106	106
query22	4014	4268	4259	4259
query23	34164	32882	33006	32882
query24	6591	2299	2426	2299
query25	514	472	453	453
query26	1213	271	157	157
query27	2324	452	333	333
query28	5581	2471	2436	2436
query29	679	562	416	416
query30	231	185	156	156
query31	945	856	770	770
query32	70	61	60	60
query33	508	355	306	306
query34	762	836	535	535
query35	804	796	756	756
query36	1070	1027	961	961
query37	120	101	83	83
query38	4318	4197	4189	4189
query39	1417	1390	1421	1390
query40	205	115	100	100
query41	53	51	54	51
query42	118	103	105	103
query43	529	541	497	497
query44	1296	811	785	785
query45	183	173	172	172
query46	874	1072	660	660
query47	1803	1793	1766	1766
query48	385	396	309	309
query49	776	494	408	408
query50	643	698	408	408
query51	4178	4163	4148	4148
query52	113	105	96	96
query53	229	265	192	192
query54	488	499	413	413
query55	82	77	78	77
query56	266	266	245	245
query57	1134	1135	1070	1070
query58	259	239	248	239
query59	3089	3149	2847	2847
query60	284	269	254	254
query61	118	116	118	116
query62	799	726	641	641
query63	227	186	191	186
query64	4181	1009	659	659
query65	3218	3158	3168	3158
query66	1068	422	311	311
query67	15904	15427	15276	15276
query68	4302	855	547	547
query69	472	302	265	265
query70	1255	1125	1154	1125
query71	393	301	265	265
query72	5835	3901	3879	3879
query73	655	772	361	361
query74	10260	9132	8794	8794
query75	3195	3154	2680	2680
query76	3141	1172	773	773
query77	505	372	280	280
query78	10246	10030	9309	9309
query79	2280	811	611	611
query80	685	533	482	482
query81	514	286	237	237
query82	215	159	130	130
query83	180	173	150	150
query84	237	90	77	77
query85	758	341	297	297
query86	403	322	287	287
query87	4490	4499	4543	4499
query88	3858	2223	2216	2216
query89	393	326	300	300
query90	1979	194	191	191
query91	131	144	110	110
query92	65	62	61	61
query93	1712	856	546	546
query94	670	410	291	291
query95	338	265	261	261
query96	494	612	284	284
query97	2795	2901	2712	2712
query98	229	213	194	194
query99	1271	1378	1259	1259
Total cold run time: 280938 ms
Total hot run time: 184829 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.39 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit dcf927d6cc8bce0a090127c9feadec36019be7e6, data reload: false

query1	0.03	0.05	0.03
query2	0.07	0.04	0.04
query3	0.23	0.07	0.07
query4	1.63	0.11	0.10
query5	0.42	0.42	0.41
query6	1.18	0.68	0.65
query7	0.02	0.02	0.01
query8	0.04	0.03	0.03
query9	0.58	0.52	0.52
query10	0.56	0.56	0.56
query11	0.14	0.10	0.10
query12	0.15	0.11	0.12
query13	0.61	0.59	0.61
query14	2.84	2.85	2.86
query15	0.89	0.82	0.83
query16	0.37	0.37	0.38
query17	1.10	1.06	1.00
query18	0.23	0.21	0.21
query19	1.98	1.86	2.01
query20	0.02	0.00	0.02
query21	15.35	0.92	0.59
query22	0.75	0.74	0.81
query23	15.22	1.48	0.65
query24	2.84	1.20	1.19
query25	0.14	0.19	0.07
query26	0.28	0.16	0.14
query27	0.08	0.05	0.04
query28	13.74	0.99	0.44
query29	12.59	3.94	3.29
query30	0.25	0.09	0.08
query31	2.81	0.59	0.38
query32	3.22	0.55	0.47
query33	2.99	3.06	3.02
query34	16.56	5.20	4.57
query35	4.56	4.61	4.58
query36	0.64	0.50	0.49
query37	0.09	0.06	0.06
query38	0.04	0.04	0.03
query39	0.03	0.03	0.03
query40	0.16	0.14	0.13
query41	0.08	0.03	0.02
query42	0.04	0.03	0.02
query43	0.03	0.03	0.03
Total cold run time: 105.58 s
Total hot run time: 31.39 s

@HappenLee
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 32526 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 4f0bb8c371030402ddef26a2b6752d28f47e1ebf, data reload: false

------ Round 1 ----------------------------------
q1	17582	5551	5415	5415
q2	2047	324	163	163
q3	10407	1364	722	722
q4	10215	963	546	546
q5	7543	2435	2184	2184
q6	185	165	131	131
q7	896	738	631	631
q8	9243	1385	1154	1154
q9	5305	4944	4948	4944
q10	6855	2344	1918	1918
q11	488	280	264	264
q12	340	361	220	220
q13	17753	3639	3115	3115
q14	247	241	201	201
q15	516	485	489	485
q16	635	621	583	583
q17	562	854	327	327
q18	7323	6457	6516	6457
q19	1220	976	558	558
q20	325	329	199	199
q21	2870	2148	1996	1996
q22	374	345	313	313
Total cold run time: 102931 ms
Total hot run time: 32526 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5533	5492	5530	5492
q2	239	329	233	233
q3	2277	2629	2282	2282
q4	1452	1857	1352	1352
q5	4300	4764	4621	4621
q6	160	155	123	123
q7	2105	2029	1836	1836
q8	2640	2828	2664	2664
q9	7302	7217	7210	7210
q10	3019	3248	2777	2777
q11	574	500	498	498
q12	670	793	588	588
q13	3550	3916	3210	3210
q14	292	305	271	271
q15	513	483	462	462
q16	627	679	658	658
q17	1220	1753	1264	1264
q18	7737	7433	7322	7322
q19	831	1058	1045	1045
q20	2013	2035	1886	1886
q21	5656	5226	5018	5018
q22	619	593	567	567
Total cold run time: 53329 ms
Total hot run time: 51379 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 184461 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 4f0bb8c371030402ddef26a2b6752d28f47e1ebf, data reload: false

query1	966	378	363	363
query2	6530	2095	2001	2001
query3	6789	217	214	214
query4	33609	23381	22987	22987
query5	4306	602	468	468
query6	272	195	177	177
query7	4599	481	313	313
query8	280	230	222	222
query9	9385	2675	2682	2675
query10	480	307	254	254
query11	18102	15511	14852	14852
query12	153	107	103	103
query13	1670	547	403	403
query14	9755	7353	6836	6836
query15	249	195	198	195
query16	7413	640	465	465
query17	1607	740	567	567
query18	1977	388	307	307
query19	218	182	162	162
query20	115	113	110	110
query21	226	123	102	102
query22	4345	4347	4195	4195
query23	33616	33148	33957	33148
query24	7023	2238	2278	2238
query25	469	449	378	378
query26	1064	250	154	154
query27	2078	472	334	334
query28	5057	2464	2442	2442
query29	525	525	421	421
query30	233	195	157	157
query31	918	873	767	767
query32	95	61	58	58
query33	501	388	304	304
query34	773	860	502	502
query35	790	803	719	719
query36	1000	1055	940	940
query37	125	110	86	86
query38	4227	4199	4068	4068
query39	1437	1402	1412	1402
query40	206	117	102	102
query41	53	53	52	52
query42	120	103	104	103
query43	513	517	477	477
query44	1330	799	802	799
query45	183	172	165	165
query46	860	1027	635	635
query47	1790	1820	1759	1759
query48	379	414	320	320
query49	784	496	403	403
query50	623	640	404	404
query51	4170	4159	4130	4130
query52	105	109	91	91
query53	228	250	197	197
query54	482	517	409	409
query55	81	75	79	75
query56	259	277	281	277
query57	1128	1108	1110	1108
query58	253	230	242	230
query59	3095	3328	2930	2930
query60	278	265	257	257
query61	140	115	114	114
query62	790	706	648	648
query63	217	186	184	184
query64	4665	1053	678	678
query65	3226	3114	3106	3106
query66	872	415	304	304
query67	15790	15371	15751	15371
query68	4326	853	555	555
query69	446	284	310	284
query70	1136	1135	1092	1092
query71	390	287	259	259
query72	5857	3829	3728	3728
query73	656	759	367	367
query74	9726	9154	8743	8743
query75	3151	3148	2650	2650
query76	3243	1174	757	757
query77	466	364	290	290
query78	10066	9888	9310	9310
query79	2827	800	600	600
query80	1557	514	456	456
query81	557	273	238	238
query82	355	152	123	123
query83	184	169	143	143
query84	236	98	77	77
query85	842	371	304	304
query86	454	331	301	301
query87	4537	4464	4395	4395
query88	4675	2192	2172	2172
query89	385	324	291	291
query90	1784	189	194	189
query91	137	133	107	107
query92	69	55	52	52
query93	2764	848	537	537
query94	714	418	296	296
query95	330	265	258	258
query96	481	609	288	288
query97	2788	2868	2724	2724
query98	228	201	196	196
query99	1271	1353	1262	1262
Total cold run time: 282235 ms
Total hot run time: 184461 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.16 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 4f0bb8c371030402ddef26a2b6752d28f47e1ebf, data reload: false

query1	0.04	0.04	0.04
query2	0.07	0.04	0.03
query3	0.24	0.08	0.06
query4	1.61	0.11	0.11
query5	0.42	0.44	0.42
query6	1.14	0.67	0.65
query7	0.03	0.02	0.02
query8	0.04	0.03	0.03
query9	0.59	0.51	0.51
query10	0.55	0.56	0.56
query11	0.14	0.10	0.10
query12	0.13	0.11	0.11
query13	0.62	0.61	0.59
query14	2.72	2.81	2.82
query15	0.88	0.82	0.84
query16	0.38	0.40	0.38
query17	1.08	1.01	1.06
query18	0.23	0.20	0.20
query19	1.98	1.80	1.98
query20	0.02	0.01	0.02
query21	15.35	0.94	0.57
query22	0.76	0.73	0.69
query23	15.31	1.46	0.59
query24	3.25	1.33	1.21
query25	0.18	0.17	0.16
query26	0.28	0.13	0.15
query27	0.06	0.05	0.05
query28	13.55	1.01	0.43
query29	12.58	3.96	3.32
query30	0.25	0.09	0.07
query31	2.82	0.60	0.38
query32	3.22	0.55	0.46
query33	2.96	2.97	3.03
query34	16.62	5.17	4.54
query35	4.64	4.55	4.55
query36	0.65	0.48	0.49
query37	0.09	0.06	0.06
query38	0.04	0.04	0.03
query39	0.03	0.02	0.02
query40	0.17	0.13	0.13
query41	0.08	0.03	0.02
query42	0.03	0.02	0.02
query43	0.03	0.02	0.03
Total cold run time: 105.86 s
Total hot run time: 31.16 s

Copy link
Contributor

github-actions bot commented Feb 6, 2025

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Feb 6, 2025
Copy link
Contributor

github-actions bot commented Feb 6, 2025

PR approved by anyone and no changes requested.

@HappenLee HappenLee merged commit f0f990b into apache:master Feb 6, 2025
25 of 26 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by one committer. reviewed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants