[Bug] Kyuubi Kubernetes Client would not reload token when service account token expired

[kennywgx]

Code of Conduct

• I agree to follow this project's Code of Conduct

Search before asking

• I have searched in the issues and found no similar issues.

Describe the bug

Fault: KubenetesClientException - Unauthorized. http code=401

My kyuubi server was running on Azure Kubenetes Service(简写：AKS) , and we met this error in below pic.

Note: Kyuubi server worked well at the beginning, the error happened after 1 hour. AKS said that it was caused by the token expiration. We suspected that kyuubi won't auto reload the serviceAccount token in the pod.

Root Cause

The AKS serviceaccount token expires in 1 hour by default. Thus, the kyuubi will have an peaceful hour after starting, but after that, the token will expires, a continuous KubenetesClientException with 401 http code will be threw out.

Kubenetes client token refresh mechanism

Dive into the code of Kubenetes Client, we found that it has already integrated a TokenRefreshInterceptor to handle the 401 http status code issue, but it doesn't work at all in kyuubi. Here's why:

Kyuubi load the serviceAccount token by itself when initializing kube client, which is NOT RECOMMENDED. In fact, the kubeclient will auto configure the desired token, and take care of the token expiration time on its own.

Check out the TokenRefreshInterceptor code in Kubenetes client repo, we can see that the refreshToken method will skip refreshing token if you have specified a oauthToken by yourself.

Fix

We simply removed the code of setting oauthToken in the kyuubi, and it works very well now.

In addition, if you want to provide some flexible ways for the the authentication configuration, consider using tokenProvider instead of a token value. I would be very willing to commit a PR if you guys agree with my opinion, thanks.

Affects Version(s)

1.7.0~1.9.3

Kyuubi Server Log Output

No response

Kyuubi Engine Log Output

No response

Kyuubi Server Configurations

No response

Kyuubi Engine Configurations

No response

Additional context

No response

Are you willing to submit PR?

• Yes. I would be willing to submit a PR with guidance from the Kyuubi community to fix.
• No. I cannot submit a PR at this time.

[github-actions]

Hello @kennywgx,
Thanks for finding the time to report the issue!
We really appreciate the community's efforts to improve Apache Kyuubi.

[kennywgx]

上面的问题，我在内存中监控了相关代码的执行结果，确定了当token需要被刷新的时候，一直没有刷新成功：

token的值一直保持不变：

观察到 refreshToken() 方法中，如果oauthToken已经存在，方法就立刻返回，不执行下面的refresh逻辑

[pan3793]

see #6883