Sidecars terminated using step signalling system

[sellitforcache]

Feature request

Remove the use of nop container to terminate sidecars. If the entrypoint script was used to override sidecar containers in general, sidecars could use the same filesystem signalling system used to sequence step containers. The entrypoint script would have to be modified to to also accept a stop signal in addition to a start/wait signal.

Use case

Removing the nop container has multiple benefits

1. Sidecar container logs persist - this means that sidecars can be more useful in e.g. the Dashboard since they can be used to monitor task processes.
2. Better supports parallelism. Having container-level parallelism avoids scheduling issues still present in pod-level parallelism sharing writeable PVCs.
3. Exit codes are consistent sidecar nop image replacement cannot seem to exit with code 0 #4889 also resolved by this change

[vdemeester]

@sellitforcache thanks for this issue. I think that's a valid point, I am not sure I remember why we didn't go the entrypoint hack way for sidecars, but I think it would definitely have some benefits.

I am not sure what you mean on 2. sidecars are not like steps (or shouldn't really be used that way), so I am a bit curious.

[sellitforcache]

@vdemeester Thanks! Would some additional design be needed, or could I raise a PR for this myself now?

2 - I want to use a sidecar to run an api monitor that prints logs reporting current status, then print results when the task steps complete. Is this still a misuse of a sidecar? My point is that often it is better to have container level parallelism since they can natively mount writable shared filesystems without inter-node issues that you get when you run parallel tasks as separate pods.

[sellitforcache]

Would also resolve #1131

[vdemeester]

Tentatively adding this to 1.0 milestone 😛