Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

support using git-sync with PVC (allowing there to only be one git-sync container) #283

Open
thesuperzapper opened this issue Jul 8, 2021 · 4 comments · May be fixed by #575
Open

support using git-sync with PVC (allowing there to only be one git-sync container) #283

thesuperzapper opened this issue Jul 8, 2021 · 4 comments · May be fixed by #575
Labels
kind/enhancement kind - new features or changes
Milestone
airflow-8.10.0

Comments

@thesuperzapper
Copy link
Member

Currently, we don't let users enable dags.gitSync and dags.persistence at the same time.

If we do this (obviously requiring dags.persistence.accessMode = ReadWriteMany), we can massively reduce the number of git-sync containers required.

This is because we won't have to put a git-sync container in every Pod, but rather have all pods mount the PVC, while having a central single git-sync deployment that keeps the PVC up to date.
@thesuperzapper thesuperzapper added the kind/enhancement kind - new features or changes label Jul 8, 2021
@justbeay
Copy link

in airflow1 version, gitSync with dags pvc works. we hope it keeps the same in airflow2. which would not leading the task container waiting for git clone done (this masters a lot if the git repo is too big or poor network connection)

@thesuperzapper
Copy link
Member Author

@justbeay two comments:

  1. version 8.X.X of the chart can be used with both airflow 1 and airflow 2 (see the Airflow Version Support table)
  2. while the 7.X.X version of the chart may have let you enable a PVC with git-sync enabled, it would have resulted in all the git-sync containers fighting over the sync

@stale stale bot added lifecycle/stale lifecycle - this is stale and removed lifecycle/stale lifecycle - this is stale labels Sep 24, 2021
@thesuperzapper thesuperzapper mentioned this issue Sep 30, 2021
Open
@stale stale bot added the lifecycle/stale lifecycle - this is stale label Nov 25, 2021
@stale stale bot closed this as completed Dec 3, 2021
@stale stale bot removed the lifecycle/stale lifecycle - this is stale label Dec 14, 2021
@thesuperzapper thesuperzapper added the lifecycle/frozen lifecycle - this can't become stale label Dec 14, 2021
@thesuperzapper thesuperzapper removed the lifecycle/frozen lifecycle - this can't become stale label Mar 22, 2022
@thesuperzapper thesuperzapper added this to the airflow-8.7.0 milestone Mar 22, 2022
@airflow-helm airflow-helm deleted a comment from stale bot Mar 22, 2022
@airflow-helm airflow-helm deleted a comment from stale bot Mar 22, 2022
@thesuperzapper
Copy link
Member Author

I think this feature is really important for KubernetesExecutor, as right now if cloning the git repo takes a long time, then EVERY task will take at least that long, this is because the git-clone init-container must run before the task can, as it needs the DAG definitions.

@karakanb karakanb linked a pull request May 4, 2022 that will close this issue
Open
6 tasks
@karakanb
Copy link
Contributor

I have created a PR trying to address these, I'd appreciate feedback on it.

@thesuperzapper thesuperzapper mentioned this issue May 13, 2023
Closed
2 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
kind/enhancement kind - new features or changes
Projects
None yet
Milestone
airflow-8.10.0
Development

Successfully merging a pull request may close this issue.

feat: add support for git-sync with PVC (central sync deployment) karakanb/charts
3 participants
@justbeay @thesuperzapper @karakanb