Make runtime parameters optional for unresolved references

[Ankitha-Krisshnan]

Is your feature request related to a problem? Please describe.

We want to be able to reference an artifact produced by a previous run of the same pipeline.

Currently, this can be achieved using pipeline dependencies:

apiVersion: pipelines.kubeflow.org/v1alpha6
kind: RunConfiguration
metadata:
  name: recursive-rc
spec:
  run:
    pipeline: ...
    artifacts:
    - name: examples
      path: ...
    runtimeParameters:
    - name: examples_location
      valueFrom:
        runConfigurationRef:
          name: recursive-rc
          outputArtifact: examples

Unfortunately, referencing an output artifact from the same RunConfiguration as a runtime parameter requires the artifact to exist before it can be referenced. This creates a challenge for the first pipeline run, in which the RunConfiguration must be deployed with only the artifact definition.

Describe the solution you'd like
We propose a way of marking some runtime parameters as optional, to handle cases where the artifact might not exist yet. Example:

...
    runtimeParameters:
    - name: examples_location
      valueFrom:
        runConfigurationRef:
          name: recursive-rc
          outputArtifact: examples
          optional: true

This would allow a Run or RunSchedule to be created even if some optional references aren't resolved, omitting those parameters. The pipeline would then detect if the parameter was passed and decide on the appropriate action (e.g., resume if an artifact is passed).

Additional context
Referencing an output artifact from a previous pipeline run in a current run as a runtime parameter is part of work to explore maintaining state between model training runs. This can be used to resume training or other use cases where new data becomes available at a later time, but where we want to avoid reprocessing old data.