[KYUUBI #5795][K8S] Support to cleanup the spark driver pod periodically #5806

liaoyt · 2023-12-02T08:59:58Z

🔍 Description

Issue References 🔗

This pull request fixes #5795

Describe Your Solution 🔧

Create a single daemon thread to traverse cache map periodically, which will evict expired cache and trigger a pod clean up operation.

Types of changes 🔖

Bugfix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to change)

Test Plan 🧪

Behavior Without This Pull Request ⚰️

Behavior With This Pull Request 🎉

Related Unit Tests

Checklists

📝 Author Self Checklist

My code follows the style guidelines of this project
I have performed a self-review
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
My changes generate no new warnings
I have added tests that prove my fix is effective or that my feature works
New and existing unit tests pass locally with my changes
This patch was not authored or co-authored using Generative Tooling

📝 Committer Pre-Merge Checklist

Be nice. Be informative.

pan3793 · 2023-12-06T09:54:07Z

kyuubi-server/src/main/scala/org/apache/kyuubi/engine/KubernetesApplicationOperation.scala

@@ -147,6 +150,26 @@ class KubernetesApplicationOperation extends ApplicationOperation with Logging {
        }
      })
      .build()
+    if (conf.get(KyuubiConf.KUBERNETES_SPARK_CLEANUP_TERMINATED_DRIVE_POD_IMMEDIATELY)) {
+      expireCleanUpTriggerCacheExecutor = Executors.newSingleThreadScheduledExecutor(


Use ThreadUtils to create the executor

pan3793 · 2023-12-06T10:00:40Z

kyuubi-common/src/main/scala/org/apache/kyuubi/config/KyuubiConf.scala

@@ -1241,6 +1241,15 @@ object KyuubiConf {
      .stringConf
      .createWithDefault(KubernetesCleanupDriverPodStrategy.NONE.toString)

+  val KUBERNETES_SPARK_CLEANUP_TERMINATED_DRIVE_POD_IMMEDIATELY: ConfigEntry[Boolean] =
+    buildConf("kyuubi.kubernetes.spark.cleanupTerminatedDriverPodImmediately")


sorry for making you back and forth, I think the configuration name is confusing now. how about?

kyuubi.kubernetes.spark.cleanupTerminatedDriverPod.kind=[NONE|ALL|COMPLETED] kyuubi.kubernetes.spark.cleanupTerminatedDriverPod.checkInterval=[PT1M]

pan3793 · 2023-12-06T10:02:46Z

kyuubi-server/src/main/scala/org/apache/kyuubi/engine/KubernetesApplicationOperation.scala

+            val keys = cleanupTerminatedAppInfoTrigger.asMap().keySet().asScala
+            for (key <- keys) {
+              // do get to trigger cache eviction
+              cleanupTerminatedAppInfoTrigger.getIfPresent(key)
+            }


Suggested change

val keys = cleanupTerminatedAppInfoTrigger.asMap().keySet().asScala

for (key <- keys) {

// do get to trigger cache eviction

cleanupTerminatedAppInfoTrigger.getIfPresent(key)

}

val keys = cleanupTerminatedAppInfoTrigger.asMap().asScala.foreach { case (key, _) =>

// do get to trigger cache eviction

cleanupTerminatedAppInfoTrigger.getIfPresent(key)

}

pan3793 · 2023-12-06T10:04:42Z

kyuubi-server/src/main/scala/org/apache/kyuubi/engine/KubernetesApplicationOperation.scala

+      expireCleanUpTriggerCacheExecutor = Executors.newSingleThreadScheduledExecutor(
+        new ThreadFactoryBuilder().setDaemon(true).setNameFormat(
+          s"pod-clean-up-trigger-thread").build())
+      expireCleanUpTriggerCacheExecutor.scheduleAtFixedRate(


prefer to use scheduleAtFixedDelay if possible

I have encountered a performance issue when the task execution time is longer than the scheduling interval, which causes high CPU usage.

codecov-commenter · 2023-12-07T03:43:56Z

Codecov Report

Attention: 4 lines in your changes are missing coverage. Please review.

Comparison is base (5b3a78d) 61.27% compared to head (75c2b68) 61.25%.
Report is 2 commits behind head on master.

Files	Patch %	Lines
...kyuubi/engine/KubernetesApplicationOperation.scala	75.00%	3 Missing and 1 partial ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##             master    #5806      +/-   ##
============================================
- Coverage     61.27%   61.25%   -0.03%     
  Complexity       23       23              
============================================
  Files           608      608              
  Lines         36027    36047      +20     
  Branches       4951     4951              
============================================
+ Hits          22076    22080       +4     
- Misses        11556    11573      +17     
+ Partials       2395     2394       -1

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

pan3793 · 2023-12-07T05:54:47Z

Merging to master/1.8

# 🔍 Description ## Issue References 🔗 This pull request fixes #5795 ## Describe Your Solution 🔧 Create a single daemon thread to traverse cache map periodically, which will evict expired cache and trigger a pod clean up operation. ## Types of changes 🔖 - [ ] Bugfix (non-breaking change which fixes an issue) - [x] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 #### Behavior Without This Pull Request ⚰️ #### Behavior With This Pull Request 🎉 #### Related Unit Tests --- # Checklists ## 📝 Author Self Checklist - [x] My code follows the [style guidelines](https://kyuubi.readthedocs.io/en/master/contributing/code/style.html) of this project - [ ] I have performed a self-review - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [ ] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective or that my feature works - [ ] New and existing unit tests pass locally with my changes - [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) ## 📝 Committer Pre-Merge Checklist - [x] Pull request title is okay. - [x] No license issues. - [x] Milestone correctly set? - [ ] Test coverage is ok - [x] Assignees are selected. - [x] Minimum number of approvals - [x] No changes are requested **Be nice. Be informative.** Closes #5806 from liaoyt/master. Closes #5795 75c2b68 [yeatsliao] cleanup driver pod periodically Authored-by: yeatsliao <[email protected]> Signed-off-by: Cheng Pan <[email protected]> (cherry picked from commit 27ad102) Signed-off-by: Cheng Pan <[email protected]>

sudohainguyen · 2024-04-16T11:24:28Z

I'm using v1.9 but this change seems not working 🤔
is there any logs to check if the daemon is still active? @liaoyt

liaoyt · 2024-04-22T07:31:39Z

I'm using v1.9 but this change seems not working 🤔 is there any logs to check if the daemon is still active? @liaoyt

I'm sorry currently there is not any logs to check if the deamon is still active.
but you can try check if there is a thread named 'pod-cleanup-trigger-thread'

sudohainguyen · 2024-04-22T07:43:59Z

~~just checked, there's no thread named "pod-cleanup-trigger-thread"~~
found it

sudohainguyen · 2024-04-22T07:51:55Z

but still doesn't delete pods

A-little-bit-of-data · 2024-07-05T01:39:05Z

pod-cleanup-trigger-thread

I use the v1.9.1 version and configure the following two parameters kyuubi.kubernetes.spark.cleanupTerminatedDriverPod.kind=ALL
kyuubi.kubernetes.terminatedApplicationRetainPeriod=PT1M
After the executor is deleted after the timeout, the driver pod is in the completed state, but it is still retained and needs to be manually deleted. I also checked the pod log, there's no thread named "pod-cleanup-trigger-thread" @liaoyt

github-actions bot added kind:documentation Documentation is a feature! module:server module:common labels Dec 2, 2023

liaoyt changed the title ~~[KYUUBI #5731][K8S] Support to cleanup the spark driver pod periodically~~ [KYUUBI #5795][K8S] Support to cleanup the spark driver pod periodically Dec 2, 2023

pan3793 reviewed Dec 6, 2023

View reviewed changes

cleanup driver pod periodically

75c2b68

liaoyt force-pushed the master branch from aa63bf6 to 75c2b68 Compare December 7, 2023 02:28

pan3793 approved these changes Dec 7, 2023

View reviewed changes

pan3793 assigned liaoyt Dec 7, 2023

pan3793 added this to the v1.8.1 milestone Dec 7, 2023

pan3793 closed this in 27ad102 Dec 7, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[KYUUBI #5795][K8S] Support to cleanup the spark driver pod periodically #5806

[KYUUBI #5795][K8S] Support to cleanup the spark driver pod periodically #5806

liaoyt commented Dec 2, 2023 •

edited by pan3793

Loading

pan3793 Dec 6, 2023

pan3793 Dec 6, 2023

liaoyt Dec 6, 2023

pan3793 Dec 6, 2023

pan3793 Dec 6, 2023

codecov-commenter commented Dec 7, 2023

pan3793 commented Dec 7, 2023

sudohainguyen commented Apr 16, 2024 •

edited

Loading

liaoyt commented Apr 22, 2024 •

edited

Loading

sudohainguyen commented Apr 22, 2024 •

edited

Loading

sudohainguyen commented Apr 22, 2024

A-little-bit-of-data commented Jul 5, 2024

[KYUUBI #5795][K8S] Support to cleanup the spark driver pod periodically #5806

[KYUUBI #5795][K8S] Support to cleanup the spark driver pod periodically #5806

Conversation

liaoyt commented Dec 2, 2023 • edited by pan3793 Loading

🔍 Description

Issue References 🔗

Describe Your Solution 🔧

Types of changes 🔖

Test Plan 🧪

Behavior Without This Pull Request ⚰️

Behavior With This Pull Request 🎉

Related Unit Tests

Checklists

📝 Author Self Checklist

📝 Committer Pre-Merge Checklist

pan3793 Dec 6, 2023

Choose a reason for hiding this comment

pan3793 Dec 6, 2023

Choose a reason for hiding this comment

liaoyt Dec 6, 2023

Choose a reason for hiding this comment

pan3793 Dec 6, 2023

Choose a reason for hiding this comment

pan3793 Dec 6, 2023

Choose a reason for hiding this comment

codecov-commenter commented Dec 7, 2023

Codecov Report

pan3793 commented Dec 7, 2023

sudohainguyen commented Apr 16, 2024 • edited Loading

liaoyt commented Apr 22, 2024 • edited Loading

sudohainguyen commented Apr 22, 2024 • edited Loading

sudohainguyen commented Apr 22, 2024

A-little-bit-of-data commented Jul 5, 2024

liaoyt commented Dec 2, 2023 •

edited by pan3793

Loading

sudohainguyen commented Apr 16, 2024 •

edited

Loading

liaoyt commented Apr 22, 2024 •

edited

Loading

sudohainguyen commented Apr 22, 2024 •

edited

Loading