Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[tempo-distributed] Ingester PVC not removed when StatefulSet is scaled down, causing the PV associated with the PVC to linger #3542

Open
HummingMind opened this issue Jan 24, 2025 · 0 comments

Comments

@HummingMind
Copy link

HummingMind commented Jan 24, 2025

Hello,
I am using the latest tempo-distributed chart v1.31.0. I installed the chart with the following settings:

ingester:
  replicas: 2
  persistence:
    enabled: true
    storageClass: name_of_my_storage_class
    size: 100Gi

Then I changed the replicas count to 1 and upgarded with the new value.

The second pod was removed, and the StatefulSet is now showing 1/1. I can see that there is no pod associated with the second PVC anymore. The issue is that the PVC for the second pod is still around. So, if the PVC is still around, the associated PV is still around and bound to it. That costs money in a public cloud.

Based on the following chart default values (which I did not specify/apply, since they are default):

ingester:
  persistentVolumeClaimRetentionPolicy:
    enabled: false
    whenScaled: Retain
    whenDeleted: Retain

the PVC should not be retained, since the retention policy is not enabled. Seems like a bug? Or am I misunerstanding something?

When I manually remove this lingering PVC, the associated PV goes into the "released" status and the cloud provider correctly removes the PV.

Thank you!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant