Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Deploy Fireweather Stations Pipeline to openshift #23

Open
franTarkenton opened this issue Jul 27, 2023 · 0 comments
Open

Deploy Fireweather Stations Pipeline to openshift #23

franTarkenton opened this issue Jul 27, 2023 · 0 comments

Comments

@franTarkenton
Copy link
Member

Recently became aware that the bcgov residency in github only has access to 20 github runners. As the load increases this will have impacts on the timing of jobs. For this reason looking to pivot to running the pipeline in openshift.

this ticket will see the python script that does the data aquisition and uploading to object storage as well as the R code that does the transformation of the xml files to csv, run in openshift.

DoD:

  • helm chart that deploys the kubernetes cron jobs
  • Investigate if the helm chart can be configured so that the two jobs have dependencies on one another. I.E. the download process runs and then calls the R script job.

To be completed, but not as part of this ticket:

  • kibana dashboard / query that allows easily identify if job succeeded or not

relates to: #21

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: Review / QA
Development

No branches or pull requests

1 participant