Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Released PVs are not deleted #452

Open
vongrad opened this issue Sep 7, 2024 · 3 comments
Open

Released PVs are not deleted #452

vongrad opened this issue Sep 7, 2024 · 3 comments

Comments

@vongrad
Copy link

vongrad commented Sep 7, 2024

I have noticed that after draining a node, the Released PVs with ReclaimPolicy set to Delete never get deleted. When running kubectl describe pv <name>, I can see the following:

Name:              pvc-3f138b3b-2c1d-4737-b53b-1e4ac53e5b92
Labels:            <none>
Annotations:       local.path.provisioner/selected-node: k3s-agent-md-ars
                   pv.kubernetes.io/provisioned-by: rancher.io/local-path
Finalizers:        [kubernetes.io/pv-protection]
StorageClass:      local-path
Status:            Released
Claim:             default/persistence-rabbitmq-cluster-server-0
Reclaim Policy:    Delete
Access Modes:      RWO
VolumeMode:        Filesystem
Capacity:          4Gi
Node Affinity:
  Required Terms:
    Term 0:        kubernetes.io/hostname in [k3s-agent-md-ars]
Message:
Source:
    Type:  LocalVolume (a persistent volume backed by local storage on a node)
    Path:  /var/lib/rancher/k3s/storage/pvc-3f138b3b-2c1d-4737-b53b-1e4ac53e5b92_default_persistence-rabbitmq-cluster-server-0
Events:
  Type     Reason              Age                   From                                                                                                Message
  ----     ------              ----                  ----                                                                                                -------
  Warning  VolumeFailedDelete  3m57s (x15 over 83m)  rancher.io/local-path_local-path-provisioner-5ccc7458d5-xlcdg_3c7bbb98-4bc1-4768-95cb-2e6f3ee20a30  failed to delete volume pvc-3f138b3b-2c1d-4737-b53b-1e4ac53e5b92: failed to delete volume pvc-3f138b3b-2c1d-4737-b53b-1e4ac53e5b92: create process timeout after 120 seconds

The node is uncordoned after few minutes and is in ready state. Could it be that the local-path gives up after 120s and never tries to clean up the released PVs? If so, can I change some config to keep trying longer?

@elimrovr
Copy link

I second this, seems like after a deployment is deleted the associated pvc/pv remains!

@dehoej
Copy link

dehoej commented Oct 28, 2024

This is also what i have experienced. Anyone found a solution?

@dehoej
Copy link

dehoej commented Oct 28, 2024

For me the issue was this, since i am using SELinux:

#420

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants