离线任务调度到kyuubi频繁出现连接超时的问题

[stgztsw]

用dolpfinScheculer做离线调度平台，通过jdbc的方式调度spark离线任务到kyuubi，所有离线任务使用同一个jdbc用户，kyuubi.engine.share.level 使用的是默认的User, 会发现dolpfinScheculer这边会频繁出现，链接超时

[ERROR] 2022-08-15 13:01:25.024 +0800 org.apache.dolphinscheduler.plugin.datasource.hive.HiveDataSourceClient:[157] - get oneSessionDataSource Connection fail SQLException: HikariPool-76 - Connection is not available, request timed out after 30077ms.
java.sql.SQLTransientConnectionException: HikariPool-76 - Connection is not available, request timed out after 30077ms.
        at com.zaxxer.hikari.pool.HikariPool.createTimeoutException(HikariPool.java:696)
        at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:197)
        at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:162)
        at com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:128)
        at org.apache.dolphinscheduler.plugin.datasource.hive.HiveDataSourceClient.getConnection(HiveDataSourceClient.java:147)
        at org.apache.dolphinscheduler.plugin.datasource.api.plugin.DataSourceClientProvider.getConnection(DataSourceClientProvider.java:74)
        at org.apache.dolphinscheduler.plugin.task.sql.SqlTask.executeFuncAndSql(SqlTask.java:185)
        at org.apache.dolphinscheduler.plugin.task.sql.SqlTask.handle(SqlTask.java:158)
        at org.apache.dolphinscheduler.server.worker.runner.TaskExecuteThread.run(TaskExecuteThread.java:180)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125)
        at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:57)
        at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.sql.SQLTimeoutException: Query timed out after 5 seconds
        at org.apache.hive.jdbc.HiveStatement.waitForOperationToComplete(HiveStatement.java:345)
        at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:251)
        at com.zaxxer.hikari.pool.PoolBase.isConnectionAlive(PoolBase.java:169)
        at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:186)
        ... 14 common frames omitted

同时在kyuubi Query Engine这边会出现大量的select 1语句发生TIMEDOUT_STATE现象。
请问这个问题该如何优化

[pan3793]

SQLException: HikariPool-76 - Connection is not available, request timed out after 30077ms.

The root cause should be HIVE-12371, we fixed it in #3152

The fixed Kyuubi JDBC driver will be delivered in v1.6.0-incubating, before that, you can build from the master branch.

[waywtdcc]

Can kyuubi 1.5.2 hive jdbc handle this?

[pan3793]

No, the fix only included in v1.6.0-incubating. #3152

[zhaohehuhu]

Do you solve this problem now? @waywtdcc