Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow task_processing-launched pods to handle their own cleanup #218

Draft
wants to merge 1 commit into
base: master
Choose a base branch
from
Draft

Allow task_processing-launched pods to handle their own cleanup #218

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 8 additions & 3 deletions task_processing/plugins/kubernetes/kube_client.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -99,12 +99,17 @@ def terminate_pod(
while attempts:
try:
logger.info(f"Attempting to terminate {pod_name}")
# NOTE: we used to force-kill pods by setting grace_period_seconds=0, but we later
# had usecases where we wanted to allow the pod to gracefully terminate (e.g., we
# have some applications that are prone to hanging due to legacy code that do not
# correctly report failures until the end of a successful execution).
#
# For the majority of our batches, the default grace period of 30s should
# reduce to a force-kill as most things will not specifically handle SIGTERMs in a
# special way :)
self.core.delete_namespaced_pod(
name=pod_name,
namespace=namespace,
# attempt to delete immediately - Pods launched by task_processing
# shouldn't need time to clean-up/drain
grace_period_seconds=0,
# this is the default, but explcitly request background deletion of releated
# objects. see:
# https://kubernetes.io/docs/concepts/workloads/controllers/garbage-collection/
Expand Down
Loading