Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Max pod name length too long #11356

Open
3 tasks done
matt-carr opened this issue Jul 14, 2023 · 13 comments
Open
3 tasks done

Max pod name length too long #11356

matt-carr opened this issue Jul 14, 2023 · 13 comments
Assignees
Labels
area/controller Controller issues, panics P1 High priority. All bugs with >=5 thumbs up that aren’t P0, plus: Any other bugs deemed high priority type/bug

Comments

@matt-carr
Copy link

matt-carr commented Jul 14, 2023

Pre-requisites

  • I have double-checked my configuration
  • I can confirm the issues exists when I tested with :latest
  • I'd like to contribute the fix myself (see contributing guide)

What happened/what you expected to happen?

Observed behaviour: When running a workflow with nested templates, if the template names are long enough the generated pod names exceed the 63 character conventional limit.

Expected behaviour: The pod names should be truncated or otherwise limited, similar to names generated by replicasets, etc.

Kubernetes documentation is a little weird on this case - https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#dns-subdomain-names says that object names should be valid dns subdomain names, which means they're limited at 253 characters (and this is the limit you use in your code, as far as I can tell from #7896 ), but the RFC it links to indicates that this is the limit for the fully qualified name, and section 6.1.3.5 Extensibility says

The DNS defines domain name syntax very generally -- a string of labels each containing up to 63 8-bit octets, separated by dots, and with a maximum total of 255

indicating that object names should, in fact, max out at 63 characters as an individual label name. Kubernetes enforcing a 63-character limit on pod names by default seems to indicate that this is the intended behaviour - trying to create a pod with a name >63 characters fails

* spec.containers[0].name: Invalid value: "foobar-work-flow-8zdl7-a-very-long-template-name-example-3607763690": must be no more than 63 characters

and pods that are created by any pod-sets (replicasets, etc) truncate appropriately

I'm not particularly fluent in Go but I think this would be simple enough to tackle on my own, if this is an acceptable issue

Version

v3.4.4

Paste a small workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflows that uses private images.

apiVersion: argoproj.io/v1alpha1
kind: WorkflowTemplate
metadata:
  name: foobar-work-flow
spec:
  entrypoint: main-workflow
  templates:
  - name: main-workflow
    steps:
      - - name: a-very-long-template-name-example
          template: a-very-long-template-name-example
  - name: a-very-long-template-name-example
    container:
        image: docker/whalesay
        command: [ cowsay ]
        args: [ "hello world" ]
        resources:
          limits:
            memory: 32Mi
            cpu: 100m

Logs from the workflow controller

(not relevant, this is the generated pod name)
foobar-work-flow-8zdl7-a-very-long-template-name-example-3607763690

Logs from in your workflow's wait container

n/a
@terrytangyuan
Copy link
Member

@isubasinghe @JPZ13 Could you help take a look?

@isubasinghe
Copy link
Member

@terrytangyuan I am on holidays atm, I can get a review done tomorrow, a bit busy tonight.

@JPZ13
Copy link
Member

JPZ13 commented Jul 20, 2023

I can review as well @terrytangyuan

@JPZ13 JPZ13 added the P3 Low priority label Jul 20, 2023
@isubasinghe isubasinghe self-assigned this Jul 25, 2023
@isubasinghe
Copy link
Member

isubasinghe commented Jul 25, 2023

I had another look at this, I was wrong, this is certainly still an issue. The names we generate are too long.
This is somewhat worrying to me.

We do generate a lot of names that are delimited by a '-', I wonder how many things would break if we changed this to a '.' That would give us access to more name length.

We should do this properly(at the source of new name generation, which likely means a lot of source code changes) instead of in the GeneratePodName function.

If anyone is reliant on the current structure of name generation (separated via dots) this would mean a breaking change.

A bit lost on what to do, any opinions @terrytangyuan?

@terrytangyuan
Copy link
Member

It's likely that users who build inhouse UIs or services rely on the names so I am a bit concerned about changing "-" to "."

This issue happens when there are nested templates. I wonder if we could generate shorter aliases pod names instead of concatenating template names.

@isubasinghe
Copy link
Member

@terrytangyuan that is a fair concern, I think it maybe possible to generate shorter aliased pod names that way.

@stale
Copy link

stale bot commented Sep 17, 2023

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

@stale stale bot added the problem/stale This has not had a response in some time label Sep 17, 2023
@terrytangyuan terrytangyuan removed the problem/stale This has not had a response in some time label Sep 20, 2023
@mikutas
Copy link
Contributor

mikutas commented Sep 21, 2023

Since Kubernetes 1.27, workflow-controller receives warning from kube-apiserver when workload names are not DNS labels.

metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must be no more than 63 characters]

https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.27.md#api-change-3
kubernetes/kubernetes#114412

@terrytangyuan
Copy link
Member

Could you paste the warning message?

@terrytangyuan
Copy link
Member

Found the specific changelog item:

Added warnings about workload resources (Pods, ReplicaSets, Deployments, Jobs, CronJobs, or ReplicationControllers) whose names are not valid DNS labels. (kubernetes/kubernetes#114412, @thockin)

@agilgur5 agilgur5 added the area/controller Controller issues, panics label Jan 29, 2024
@Garett-MacGowan
Copy link
Contributor

+1 on this. I just ran into this issue.

@agilgur5 agilgur5 added P1 High priority. All bugs with >=5 thumbs up that aren’t P0, plus: Any other bugs deemed high priority and removed P3 Low priority labels Jul 2, 2024
@rcontreras-te
Copy link

We just ran into this. Is this actually being worked on? Our current workaround is to simply rework the workflow so as to use shorter names when using nested templates. Are there any other suggestions?

@tooptoop4
Copy link
Contributor

when u say issue what breaks? it seems like harmless warning log

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/controller Controller issues, panics P1 High priority. All bugs with >=5 thumbs up that aren’t P0, plus: Any other bugs deemed high priority type/bug
Projects
None yet
Development

No branches or pull requests

9 participants