Failed to create cache java.lang.InterruptedException #334

dragenet · 2024-08-09T16:26:58Z

Jenkins and plugins versions report

Environment

Jenkins: 2.462.1
OS: Linux - 5.15.0-117-generic
Java: 21.0.4 - Eclipse Adoptium (OpenJDK 64-Bit Server VM)
---
active-directory:2.36
antisamy-markup-formatter:162.v0e6ec0fcfcf6
apache-httpcomponents-client-4-api:4.5.14-208.v438351942757
asm-api:9.7-33.v4d23ef79fcc8
atlassian-bitbucket-server-integration:4.0.0
authentication-tokens:1.119.v50285141b_7e1
blueocean:1.27.14
blueocean-bitbucket-pipeline:1.27.14
blueocean-commons:1.27.14
blueocean-config:1.27.14
blueocean-core-js:1.27.14
blueocean-dashboard:1.27.14
blueocean-display-url:2.4.3
blueocean-events:1.27.14
blueocean-git-pipeline:1.27.14
blueocean-github-pipeline:1.27.14
blueocean-i18n:1.27.14
blueocean-jwt:1.27.14
blueocean-personalization:1.27.14
blueocean-pipeline-api-impl:1.27.14
blueocean-pipeline-editor:1.27.14
blueocean-pipeline-scm-api:1.27.14
blueocean-rest:1.27.14
blueocean-rest-impl:1.27.14
blueocean-web:1.27.14
bootstrap5-api:5.3.3-1
bouncycastle-api:2.30.1.78.1-248.ve27176eb_46cb_
branch-api:2.1178.v969d9eb_c728e
caffeine-api:3.1.8-133.v17b_1ff2e0599
checks-api:2.2.0
cloudbees-bitbucket-branch-source:888.v8e6d479a_1730
cloudbees-folder:6.942.vb_43318a_156b_2
command-launcher:115.vd8b_301cc15d0
commons-lang3-api:3.14.0-76.vda_5591261cfe
commons-text-api:1.12.0-129.v99a_50df237f7
configuration-as-code:1836.vccda_4a_122a_a_e
credentials:1371.vfee6b_095f0a_3
credentials-binding:681.vf91669a_32e45
display-url-api:2.204.vf6fddd8a_8b_e9
durable-task:568.v8fb_5c57e8417
echarts-api:5.5.0-1
eddsa-api:0.3.0-4.v84c6f0f4969e
favorite:2.221.v19ca_666b_62f5
folder-properties:1.2.1
font-awesome-api:6.5.2-1
git:5.3.0
git-client:5.0.0
github:1.40.0
github-api:1.321-468.v6a_9f5f2d5a_7e
github-branch-source:1793.v1831e9c68d77
gson-api:2.11.0-41.v019fcf6125dc
handy-uri-templates-2-api:2.1.8-30.v7e777411b_148
hashicorp-vault-plugin:368.v48134f694db_f
htmlpublisher:1.36
instance-identity:185.v303dc7c645f9
ionicons-api:74.v93d5eb_813d5f
jackson2-api:2.17.0-379.v02de8ec9f64c
jakarta-activation-api:2.1.3-1
jakarta-mail-api:2.1.3-1
javadoc:280.v050b_5c849f69
javax-activation-api:1.2.0-7
javax-mail-api:1.6.2-10
jaxb:2.3.9-1
jdk-tool:80.v8a_dee33ed6f0
jenkins-design-language:1.27.14
jjwt-api:0.11.5-112.ve82dfb_224b_a_d
jobcacher:481.v15f51ca_4c6b_7
joda-time-api:2.12.7-29.v5a_b_e3a_82269a_
jquery:1.12.4-1
jquery3-api:3.7.1-2
jsch:0.2.16-86.v42e010d9484b_
json-api:20240303-41.v94e11e6de726
json-path-api:2.9.0-58.v62e3e85b_a_655
junit:1284.vf75d778f98c5
kubernetes:4280.vd919fa_528c7e
kubernetes-client-api:6.10.0-240.v57880ce8b_0b_2
kubernetes-credentials:174.va_36e093562d9
mailer:472.vf7c289a_4b_420
matrix-auth:3.2.2
matrix-project:832.va_66e270d2946
maven-plugin:3.23
metrics:4.2.21-451.vd51df8df52ec
mina-sshd-api-common:2.13.1-117.v2f1a_b_66ff91d
mina-sshd-api-core:2.13.1-117.v2f1a_b_66ff91d
okhttp-api:4.11.0-172.vda_da_1feeb_c6e
pipeline-build-step:540.vb_e8849e1a_b_d8
pipeline-graph-analysis:216.vfd8b_ece330ca_
pipeline-groovy-lib:727.ve832a_9244dfa_
pipeline-input-step:495.ve9c153f6067b_
pipeline-milestone-step:119.vdfdc43fc3b_9a_
pipeline-model-api:2.2205.vc9522a_9d5711
pipeline-model-definition:2.2205.vc9522a_9d5711
pipeline-model-extensions:2.2205.vc9522a_9d5711
pipeline-rest-api:2.34
pipeline-stage-step:312.v8cd10304c27a_
pipeline-stage-tags-metadata:2.2205.vc9522a_9d5711
pipeline-stage-view:2.34
plain-credentials:183.va_de8f1dd5a_2b_
plugin-util-api:4.1.0
prism-api:1.29.0-15
prometheus:780.v7c50a_d288424
pubsub-light:1.18
scm-api:696.v778d637b_a_762
script-security:1341.va_2819b_414686
sectioned-view:1.27
slack:734.v7f9ec8b_66975
snakeyaml-api:2.2-121.v5a_68b_9300b_d4
sse-gateway:1.27
ssh-credentials:343.v884f71d78167
sshd:3.330.vc866a_8389b_58
structs:338.v848422169819
token-macro:400.v35420b_922dcb_
trilead-api:2.147.vb_73cc728a_32e
variant:60.v7290fc0eb_b_cd
workflow-aggregator:600.vb_57cdd26fdd7
workflow-api:1322.v857eeeea_9902
workflow-basic-steps:1058.vcb_fc1e3a_21a_9
workflow-cps:3922.va_f73b_7c4246b_
workflow-durable-task-step:1364.v2fd76fb_6fd41
workflow-job:1436.vfa_244484591f
workflow-multibranch:795.ve0cb_1f45ca_9a_
workflow-scm-step:427.v4ca_6512e7df1
workflow-step-api:678.v3ee58b_469476
workflow-support:920.v59f71ce16f04
xray-connector:2.6.1

What Operating System are you using (both controller, and any agents involved in the problem)?

I'm using Jenkins on Kubernetes:

controller image: jenkins/[email protected]
agent image: jenkins/[email protected]_f6e922152-2

Reproduction steps

Execute pipeline caching nested node_modules (monorepo), cache config like the following

cache(maxCacheSize: 10000, defaultBranch: 'master', caches:[
            arbitraryFileCache(
                cacheName: 'yarn_install_cache',
                path: './',
                includes: '**/node_modules/**/*,.yarn/install-state.gz',
                excludes: '**/node_modules/@repo/**/*',
                cacheValidityDecidingFile: 'yarn.lock',
                compressionMethod: 'TAR_ZSTD'
            )
        ]) {
            sh 'yarn install'
        }

Expected Results

Cache created

Actual Results

[Cache for ./ (yarn_install_cache) with id 0374ec857381230fa83049643ce3cdee] Failed to create cache
java.lang.InterruptedException
	at java.base/java.lang.Object.wait0(Native Method)
	at java.base/java.lang.Object.wait(Unknown Source)
	at hudson.remoting.Request.call(Request.java:177)
	at hudson.remoting.Channel.call(Channel.java:1041)
	at hudson.FilePath.act(FilePath.java:1229)
	at hudson.FilePath.act(FilePath.java:1218)
	at PluginClassLoader for jobcacher//jenkins.plugins.jobcacher.arbitrary.TarArbitraryFileCacheStrategy.compress(TarArbitraryFileCacheStrategy.java:40)
	at PluginClassLoader for jobcacher//jenkins.plugins.jobcacher.arbitrary.AbstractCompressingArbitraryFileCacheStrategy.cache(AbstractCompressingArbitraryFileCacheStrategy.java:19)
	at PluginClassLoader for jobcacher//jenkins.plugins.jobcacher.ArbitraryFileCache$SaverImpl.save(ArbitraryFileCache.java:404)
	at PluginClassLoader for jobcacher//jenkins.plugins.jobcacher.CacheManager.save(CacheManager.java:98)
	at PluginClassLoader for jobcacher//jenkins.plugins.jobcacher.pipeline.CacheStepExecution$ExecutionCallback.complete(CacheStepExecution.java:103)
	at PluginClassLoader for jobcacher//jenkins.plugins.jobcacher.pipeline.CacheStepExecution$ExecutionCallback.onSuccess(CacheStepExecution.java:80)
	at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.CpsBodyExecution$SuccessAdapter.receive(CpsBodyExecution.java:375)
	at PluginClassLoader for workflow-cps//com.cloudbees.groovy.cps.Outcome.resumeFrom(Outcome.java:70)
	at PluginClassLoader for workflow-cps//com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:144)
	at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.SandboxContinuable.access$001(SandboxContinuable.java:17)
	at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:49)
	at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:180)
	at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:423)
	at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:331)
	at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:295)
	at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService.lambda$wrap$4(CpsVmExecutorService.java:136)
	at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
	at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:139)
	at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
	at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:68)
	at jenkins.util.ErrorLoggingExecutorService.lambda$wrap$0(ErrorLoggingExecutorService.java:51)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
	at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$1.call(CpsVmExecutorService.java:53)
	at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$1.call(CpsVmExecutorService.java:50)
	at org.codehaus.groovy.runtime.GroovyCategorySupport$ThreadCategoryInfo.use(GroovyCategorySupport.java:136)
	at org.codehaus.groovy.runtime.GroovyCategorySupport.use(GroovyCategorySupport.java:275)
	at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService.lambda$categoryThreadFactory$0(CpsVmExecutorService.java:50)
	at java.base/java.lang.Thread.run(Unknown Source)

Anything else?

It happens not for each execution but often no matter which compression implementation was selected

Are you interested in contributing a fix?

No response

The text was updated successfully, but these errors were encountered:

ovcharenko-di · 2025-01-29T14:48:06Z

@dragenet it seems that the exception is thrown exactly after 5 minutes, according to the log of my job. However, I could not find where the timeout is set.

jonesbusy · 2025-01-29T15:10:01Z

Not on this plugin for sure

It's cause by runNextChunk(CpsThread.java:180)

There is an hardcoded timeout for CPS Thread : https://github.com/jenkinsci/workflow-cps-plugin/blob/5b32b1a2e5b83a23354bbc147037cbdbc9d9061f/plugin/src/main/java/org/jenkinsci/plugins/workflow/cps/CpsThread.java#L176

Normally for long operation (like 'sh') Jenkins use the durable task. But for other step there seems to have this hardcoded timeout.

I don't know if we can bypass it to be honnest.

Probably the plugin need to implement in some way: https://github.com/jenkinsci/durable-task-plugin to allow the cache step to be durable (and also survive controller restart)

I never did anything like this to be honnest, so if anyone want to give some helps, I'm more than happy to review any pull request

timja · 2025-02-04T11:22:24Z

Similar error here: jenkinsci/bom#4351 (comment)

timja · 2025-02-04T11:24:12Z

@jglick got any advice on a good approach here?

jglick · 2025-02-10T19:26:18Z

I think you are looking for https://javadoc.jenkins.io/plugin/workflow-step-api/org/jenkinsci/plugins/workflow/steps/GeneralNonBlockingStepExecution.html

timja · 2025-02-10T21:52:32Z

Hmm its already using that:

jobcacher-plugin/src/main/java/jenkins/plugins/jobcacher/pipeline/CacheStepExecution.java

Line 18 in 1b30b69

public class CacheStepExecution extends GeneralNonBlockingStepExecution {

but its times out at 5 mins.

timja · 2025-02-10T21:55:04Z

Maybe this? https://javadoc.jenkins.io/plugin/workflow-step-api/org/jenkinsci/plugins/workflow/steps/SynchronousNonBlockingStepExecution.html

edit hmm not sure I think maybe not

repolevedavaj · 2025-02-10T22:38:02Z

I believe the issue is not the CacheStepExecution but the ExecutionCallback which is finally doing the cache save. It seems that this callback is executed on the CpsThread and therefore limited by the hard coded timeout. Maybe there is a way to execute the save async too.

repolevedavaj · 2025-02-10T22:41:19Z

Actually is it even documented to be used like this: Initiate background work that should not block the CPS VM thread. Call this from a CPS VM thread, such as from start or BodyExecutionCallback. onSuccess

repolevedavaj · 2025-02-10T22:49:39Z

I created a commit with this change. Probably it would make sense to checkout the implementation of org.jenkinsci.plugins.workflow.steps.GeneralNonBlockingStepExecution.TailCall#onSuccess in more detail to see whether we can improve it further, but I think the change is worth a try. I hope I am not completely wrong though...

dragenet added the bug label Aug 9, 2024

dduportal mentioned this issue Jan 30, 2025

Instability of artifact-caching-proxy jenkins-infra/helpdesk#4442

Open

jonesbusy mentioned this issue Feb 4, 2025

Job cacher PoC jenkinsci/bom#4351

Draft

6 tasks

repolevedavaj linked a pull request Feb 10, 2025 that will close this issue

Do not execute cache step body callback on CPS VM thread #396

Draft

6 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Failed to create cache java.lang.InterruptedException #334

Failed to create cache java.lang.InterruptedException #334

dragenet commented Aug 9, 2024 •

edited by timja

Loading

ovcharenko-di commented Jan 29, 2025

jonesbusy commented Jan 29, 2025

timja commented Feb 4, 2025

timja commented Feb 4, 2025

jglick commented Feb 10, 2025

timja commented Feb 10, 2025

timja commented Feb 10, 2025 •

edited

Loading

repolevedavaj commented Feb 10, 2025

repolevedavaj commented Feb 10, 2025

repolevedavaj commented Feb 10, 2025 •

edited

Loading

Failed to create cache java.lang.InterruptedException #334

Failed to create cache java.lang.InterruptedException #334

Comments

dragenet commented Aug 9, 2024 • edited by timja Loading

Jenkins and plugins versions report

What Operating System are you using (both controller, and any agents involved in the problem)?

Reproduction steps

Expected Results

Actual Results

Anything else?

Are you interested in contributing a fix?

ovcharenko-di commented Jan 29, 2025

jonesbusy commented Jan 29, 2025

timja commented Feb 4, 2025

timja commented Feb 4, 2025

jglick commented Feb 10, 2025

timja commented Feb 10, 2025

timja commented Feb 10, 2025 • edited Loading

repolevedavaj commented Feb 10, 2025

repolevedavaj commented Feb 10, 2025

repolevedavaj commented Feb 10, 2025 • edited Loading

dragenet commented Aug 9, 2024 •

edited by timja

Loading

timja commented Feb 10, 2025 •

edited

Loading

repolevedavaj commented Feb 10, 2025 •

edited

Loading