From 7d87a94dd77f43120701e48a371324a4f5f2064b Mon Sep 17 00:00:00 2001 From: panbingkun Date: Tue, 26 Mar 2024 09:53:21 -0700 Subject: [PATCH] [MINOR][CORE] When failed to canceling the job group, add a warning log ### What changes were proposed in this pull request? The pr aims to add add a warning log when failed to canceling the job group. ### Why are the changes needed? In production, we found that when there are many jobs, if we specify `a just-submitted job group` and `cancel` it using the API `SparkContext#cancelJobGroup(groupId: String)`, the cancellation will fail (`silently`) because the job has not yet entered `activeJobs`, and there is `no warning` prompt. (PS: Due to historical reasons, we are not currently using the newer API `SparkContext#cancelJobGroupAndFutureJobs(groupId: String)`) I propose to add `a warning log` here to help troubleshoot `issues`. ### Does this PR introduce _any_ user-facing change? Yes, only for log. ### How was this patch tested? N/A ### Was this patch authored or co-authored using generative AI tooling? No. Closes #45722 from panbingkun/handleJobGroupCancelled_add_log. Authored-by: panbingkun Signed-off-by: Dongjoon Hyun --- .../main/scala/org/apache/spark/scheduler/DAGScheduler.scala | 3 +++ 1 file changed, 3 insertions(+) diff --git a/core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala b/core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala index 6323052994b6a..b4333d5533c53 100644 --- a/core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala +++ b/core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala @@ -1220,6 +1220,9 @@ private[spark] class DAGScheduler( _.getProperty(SparkContext.SPARK_JOB_GROUP_ID) == groupId } } + if (activeInGroup.isEmpty && !cancelFutureJobs) { + logWarning(s"Failed to cancel job group $groupId. Cannot find active jobs for it.") + } val jobIds = activeInGroup.map(_.jobId) jobIds.foreach(handleJobCancellation(_, Option("part of cancelled job group %s".format(groupId))))