Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

RangerSparkExtension - ConfigMaps not deleted on Kubernetes #5237

Open
2 of 4 tasks
maron546 opened this issue Sep 1, 2023 · 2 comments · May be fixed by #6700
Open
2 of 4 tasks

RangerSparkExtension - ConfigMaps not deleted on Kubernetes #5237

maron546 opened this issue Sep 1, 2023 · 2 comments · May be fixed by #6700
Labels
kind:bug This is a clearly a bug priority:major

Comments

@maron546
Copy link

maron546 commented Sep 1, 2023

Code of Conduct

Search before asking

  • I have searched in the issues and found no similar issues.

Describe the bug

On Kubernetes, after each query, even if all the pods are deleted successfully, a ConfigMap still remains without related resources. The ConfigMap is named:

spark-exec-...-conf-map

and contains the following configuration files:

ranger-spark-audit.xml
ranger-spark-security.xml

Affects Version(s)

1.7.1

Kyuubi Server Log Output

Not applicable

Kyuubi Engine Log Output

Not applicable

Kyuubi Server Configurations

kyuubi.authentication=KERBEROS
kyuubi.frontend.connection.url.use.hostname=false
kyuubi.frontend.protocols=THRIFT_BINARY
kyuubi.ha.addresses=xxx
kyuubi.ha.namespace=kyuubi
kyuubi.kinit.principal=xxx
kyuubi.kinit.keytab=xxx
kyuubi.kubernetes.authenticate.caCertFile=xxx
kyuubi.kubernetes.authenticate.oauthTokenFile=xxx
kyuubi.kubernetes.namespace=xxx
kyuubi.metadata.store.jdbc.database.schema.init=false
kyuubi.metrics.enabled=true
spark.master=xxx
spark.driver.memory=xxx
spark.executor.memory=xxx
spark.dynamicAllocation.enabled=true
spark.dynamicAllocation.initialExecutors=1
spark.dynamicAllocation.maxExecutors=xxx
spark.dynamicAllocation.shuffleTracking.enabled=true
spark.sql.adaptive.enabled=true
spark.kubernetes.container.image=xxx
spark.kubernetes.container.image.pullPolicy=xxx
spark.kubernetes.driver.annotation.app.kubernetes.io/component=kyuubi
spark.kubernetes.driver.request.cores=xxx
spark.kubernetes.driver.limit.cores=xxx
spark.kubernetes.driverEnv.HADOOP_CONF_DIR=/opt/kyuubi/conf
spark.kubernetes.executor.annotation.app.kubernetes.io/component=kyuubi
spark.kubernetes.executor.request.cores=xxx
spark.kubernetes.executor.limit.cores=xxx
spark.kubernetes.executor.secrets.kyuubi-configs=/opt/kyuubi/conf
spark.executorEnv.HADOOP_CONF_DIR=/opt/kyuubi/conf
spark.kubernetes.namespace=xxx
spark.kerberos.access.hadoopFileSystems=xxx
spark.jars=/opt/ranger-plugin/libext/*.jar
spark.sql.extensions=org.apache.kyuubi.plugin.spark.authz.ranger.RangerSparkExtension

Kyuubi Engine Configurations

Not applicable

Additional context

No response

Are you willing to submit PR?

  • Yes. I would be willing to submit a PR with guidance from the Kyuubi community to fix.
  • No. I cannot submit a PR at this time.
@maron546 maron546 added kind:bug This is a clearly a bug priority:major labels Sep 1, 2023
@github-actions
Copy link

github-actions bot commented Sep 1, 2023

Hello @maron546,
Thanks for finding the time to report the issue!
We really appreciate the community's efforts to improve Apache Kyuubi.

Madhukar525722 added a commit to Madhukar525722/kyuubi that referenced this issue Sep 17, 2024
@Madhukar525722
[KYUUBI apache#5237] ConfigMaps deletion on Kubernetes
340c3d7
@Madhukar525722 Madhukar525722 linked a pull request Sep 17, 2024 that will close this issue
Open
4 tasks
@Madhukar525722
Copy link
Contributor

Hi @maron546, Good day!!!
Actually, I wanted to know what spark version you were using, which resulted in the above issue. Is it still persistent?

Madhukar525722 added a commit to Madhukar525722/kyuubi that referenced this issue Oct 5, 2024
@Madhukar525722
[KYUUBI apache#5237] ConfigMaps deletion on Kubernetes
c729e2d
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
kind:bug This is a clearly a bug priority:major
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[KYUUBI #5237] ConfigMaps deletion on Kubernetes Madhukar525722/kyuubi
2 participants
@maron546 @Madhukar525722