Template not found on initial creation

[drew-viles]

Pre-requisites

• I have double-checked my configuration
• I can confirm the issues exists when I tested with :latest
• I'd like to contribute the fix myself (see contributing guide)

What happened/what you expected to happen?

Creating a CronWorkflow fails to find 2nd template in a sequence of steps on initial creation - more details below.

Templates should be detected on first creation and shouldn't need replacing twice.

Version

I'm using helm chart: 0.24.0 - the latest available which gives me AppVersion: v3.4.7

Paste a small workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflows that uses private images.

I've created a CronWorkflow and WorkflowTemplates as follows:

apiVersion: argoproj.io/v1alpha1
kind: CronWorkflow
metadata:
  name: test-cron
  namespace: argo-workflows
spec:
  schedule: "* * * * *"
  concurrencyPolicy: "Replace"
  startingDeadlineSeconds: 0
  workflowSpec:
    entrypoint: test
    templates:
      - name: test
        steps:
          - - name: builder-test
              templateRef:
                name: test1
                template: test
          - - name: scanner-test
              templateRef:
                name: test2
                template: test
---
apiVersion: argoproj.io/v1alpha1
kind: WorkflowTemplate
metadata:
  name: test1
  namespace: argo-workflows
spec:
  entrypoint: test
  templates:
    - name: test
      script:
        image: ubuntu:22.04
        imagePullPolicy: IfNotPresent
        command: [bash]
        source: | 
          echo "test build complete"
---
apiVersion: argoproj.io/v1alpha1
kind: WorkflowTemplate
metadata:
  name: test2
  namespace: argo-workflows
spec:
  entrypoint: test
  templates:
    - name: test
      script:
        image: ubuntu:22.04
        imagePullPolicy: IfNotPresent
        command: [bash]
        source: |
          echo "test scan complete"

And I get the following error in the log (Below)

If I then change the test2 name in CronWorkflow:

          - - name: scanner-test
              templateRef:
                name: test3
                template: test

Apply that and then switch it back to:

          - - name: scanner-test
              templateRef:
                name: test2
                template: test

and apply it again.

It all works fine from that point onwards... It's a bit strange.

Logs from the workflow controller

time="2023-04-13T11:28:38.031Z" level=error msg="cannot validate Workflow: templates.test.steps[1].scanner-test template reference test2.test not found" conditionType=SpecError namespace=argo-workflows workflow=test-cron
time="2023-04-13T11:28:38.031Z" level=error msg="invalid cron workflow" cronWorkflow=argo-workflows/test-cron error="cannot validate Workflow: templates.test.steps[1].scanner-test template reference test2.test not found"

Logs from in your workflow's wait container

N/A

[drew-viles]

I've had an epiphany this morning in that this is probably a race condition. So if I apply the above example it prevents the cron working due to the error mentioned.

If I put the cron below the templates, it works instantly.

I left it for a while and the reconciliation loop came around and checked again which eventually kicks it into action.

What might be good here as an improvement to UX is an addition to the log that says "Template not found, will check again in X minutes?".

I just feel that it could cause less confusion with a template not being found when the user can see it there.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. If this is a mentoring request, please provide an update here. Thank you for your contributions.

[tooptoop4]

@drew-viles did u try create the cron after the templates?

[drew-viles]

Yeah that worked for me. Basically it's just an error that occurs if the order of deployment is cron -> templates. If it's templates -> cron then it works fine.