Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update Argo Workflows to latest version #1639

Merged
merged 25 commits into from
Apr 26, 2023
Merged
Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
25 commits
Select commit Hold shift + click to select a range
bcabe14
update and fix argo workflows
Feb 1, 2023
d65159e
update documentation
Feb 1, 2023
70c8357
small wording change
Feb 1, 2023
8abca86
[pre-commit.ci] Apply automatic pre-commit fixes
pre-commit-ci[bot] Feb 1, 2023
d0589f2
fix spacing
Feb 1, 2023
508844a
add crds
Feb 21, 2023
f5ddf15
remove unneeded service account
Feb 21, 2023
1e3565d
remove helm annotations
Feb 21, 2023
5de7390
Merge branch 'develop' into improve_argo_wf
Mar 2, 2023
2fef17e
remove CRDs and let helm chart manage them
Apr 4, 2023
e6256d1
fix upgrade command to look for nebari_version instead of qhub_version
Apr 4, 2023
e1345a5
fix a few other mentions of qhub_version
Apr 4, 2023
a36e68c
only use nebari_version since the upgrade steps should fix it
Apr 4, 2023
a47315b
add error message asking to manually update the qhub_version tag and …
Apr 4, 2023
7669972
Merge branch 'fix_upgrade_step' into improve_argo_wf
Apr 4, 2023
8717732
remove breakpoint
Apr 4, 2023
740298e
added upgrade advising users to delete Argo CRDs
Apr 5, 2023
8571e04
Merge branch 'develop' into improve_argo_wf
Adam-D-Lewis Apr 24, 2023
0ec265a
add helm chart delete command
Adam-D-Lewis Apr 24, 2023
e0256fe
Revert "add helm chart delete command"
Adam-D-Lewis Apr 24, 2023
717b8ec
add links to docs
Adam-D-Lewis Apr 25, 2023
0c568d9
change upgrade version
Adam-D-Lewis Apr 25, 2023
b934eb3
Update nebari/upgrade.py
Adam-D-Lewis Apr 25, 2023
ec13097
exit rather than throw error
Adam-D-Lewis Apr 25, 2023
ecf729b
fix tests
Adam-D-Lewis Apr 26, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 9 additions & 15 deletions docs/source/admin_guide/argo-workflows.md
Original file line number Diff line number Diff line change
@@ -1,26 +1,18 @@
# Argo Workflows
iameskild marked this conversation as resolved.
Show resolved Hide resolved

Argo Workflows is an open source container-native workflow engine for orchestrating parallel jobs on Kubernetes. Argo
workflows comes enabled by default with Qhub deployments.
workflows comes enabled by default with Nebari deployments.

## Accessing Argo Server

If Argo Workflows is enabled, users can access argo workflows server at: `your-qhub-domain.com/argo`. Log in via
If Argo Workflows is enabled, users can access argo workflows server at: `your-nebari-domain.com/argo`. Log in via
Keycloak with your usual credentials.

Refer to the [Argo documentation](https://argoproj.github.io/argo-workflows/) for further details on Argo Workflows.

## Submitting a workflow via Argo Server

You can submit a workflow by clicking "SUBMIT NEW WORKFLOW" on the landing page.

![See Argo Server Landing Page](../images/argo-server-landing-page.png)

## Overrides of Argo Workflows Helm Chart values

Argo Workflows is deployed using Argo Workflows Helm Chart version 0.13.1. The values.yaml for the helm chart can be
overridden as needed via the overrides flag. The default values file can be found
[here](https://github.com/argoproj/argo-helm/blob/argo-workflows-0.13.1/charts/argo-workflows/values.yaml). For example,
Argo Workflows is deployed using the Argo Workflows Helm Chart. The values.yaml for the helm chart can be overridden as
needed via the overrides flag. The default values file can be found
[here](https://github.com/argoproj/argo-helm/blob/argo-workflows-0.22.9/charts/argo-workflows/values.yaml). For example,
the following could be done to add additional environment variables to the controller container.

```yaml
Expand All @@ -35,10 +27,12 @@ argo_workflows:

## Disabling Argo Workflows

To turn off the cluster monitoring on QHub deployments, simply turn off the feature flag within your `qhub-config.yaml`
file. For example:
To turn off the cluster monitoring on Nebari deployments, simply turn off the feature flag within your
`nebari-config.yaml` file. For example:
Comment on lines +30 to +31
Copy link
Member Author

@Adam-D-Lewis Adam-D-Lewis Feb 2, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should say "To disable Argo Workflows"


```yaml
argo_workflows:
enabled: false
```

Refer to the [Argo documentation](https://argoproj.github.io/argo-workflows/) for further details on Argo Workflows.
Binary file added docs/source/images/argo-workflows-user-tab.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
32 changes: 32 additions & 0 deletions docs/source/user_guide/argo_workflows.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
# Argo Workflows

Argo Workflows is an open source container-native workflow engine for orchestrating parallel jobs on Kubernetes. Argo
workflows comes enabled by default with Nebari deployments.

## Accessing Argo Server

If Argo Workflows is enabled, users can access argo workflows server at: `your-nebari-domain.com/argo`. Log in via
Keycloak with your usual credentials.

## Submitting a Workflow via Argo Server

You can submit a workflow by clicking "SUBMIT NEW WORKFLOW" on the landing page assuming you have the appropriate
permissions.

![Argo Server Landing Page](../images/argo-server-landing-page.png)

## Submitting a workflow via Argo CLI

You can submit or manage workflows via the Argo CLI. The Argo CLI can be downloaded from the
[Argo Releases](https://github.com/argoproj/argo-workflows/releases) page. After downloading the CLI, you can get your
Comment on lines +20 to +21
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This can be done outside of Nebari. I'm not sure if we want to provide some better solution to install the CLI tool inside Nebari. It's not available via conda that I could find.

Copy link
Member Author

@Adam-D-Lewis Adam-D-Lewis Feb 8, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note: Users can download the binary from the releases page and use it inside of Nebari as well

token from the Argo Server UI by clicking on the user tab in the bottom left corner and then clicking "Copy To
Clipboard". You'll need to make a few edits to access to what was copied for Argo CLI to work correctly. The base href
should be `ARGO_BASE_HREF=/argo` in the default nebari installation and you need to set the namespace where Argo was
deployed (dev by default) `ARGO_NAMESPACE=dev`. After setting those variables and the others copied from the Argo Server
UI, you can check that things are working by running `argo list`.

![Argo Workflows User Tab](../images/argo-workflows-user-tab.png)

## Additional Argo Workflows Resources

Refer to the [Argo documentation](https://argoproj.github.io/argo-workflows/) for further details on Argo Workflows.
Original file line number Diff line number Diff line change
Expand Up @@ -8,16 +8,21 @@ resource "helm_release" "argo-workflows" {
namespace = var.namespace
repository = "https://argoproj.github.io/argo-helm"
chart = "argo-workflows"
version = "0.13.1"
version = "0.22.9"

values = concat([
file("${path.module}/values.yaml"),
# https://github.com/argoproj/argo-helm/blob/argo-workflows-0.13.1/charts/argo-workflows/values.yaml


jsonencode({
singleNamespace = true # Restrict Argo to operate only in a single namespace (the namespace of the Helm release)

workflow = {
serviceAccount = {
create = true
name = "argo-workflow"
}
}

controller = {
metricsConfig = {
enabled = true # enable prometheus
Expand All @@ -28,6 +33,11 @@ resource "helm_release" "argo-workflows" {
nodeSelector = {
"${var.node-group.key}" = var.node-group.value
}
workflowDefaults = {
spec = {
serviceAccountName = "argo-workflow"
}
}
}

server = {
Expand Down Expand Up @@ -62,9 +72,6 @@ resource "helm_release" "argo-workflows" {
"${var.node-group.key}" = var.node-group.value
}
}
controller = {
iameskild marked this conversation as resolved.
Show resolved Hide resolved
containerRuntimeExecutor = "emissary"
}

})
], var.overrides)
Expand Down Expand Up @@ -180,9 +187,9 @@ resource "kubernetes_cluster_role_binding" "argo-admin-rb" {
}
}

resource "kubernetes_service_account" "argo-edit-sa" {
resource "kubernetes_service_account" "argo-dev-sa" {
metadata {
name = "argo-edit"
name = "argo-dev"
namespace = var.namespace
annotations = {
"workflows.argoproj.io/rbac-rule" : "'argo_developer' in groups"
Expand All @@ -192,9 +199,9 @@ resource "kubernetes_service_account" "argo-edit-sa" {
}
}

resource "kubernetes_cluster_role_binding" "argo-edit-rb" {
resource "kubernetes_cluster_role_binding" "argo-dev-rb" {
metadata {
name = "argo-edit"
name = "argo-dev"
}

role_ref {
Expand All @@ -204,10 +211,12 @@ resource "kubernetes_cluster_role_binding" "argo-edit-rb" {
}
subject {
kind = "ServiceAccount"
name = kubernetes_service_account.argo-edit-sa.metadata.0.name
name = kubernetes_service_account.argo-dev-sa.metadata.0.name
namespace = var.namespace
}
}


resource "kubernetes_service_account" "argo-view-sa" {
metadata {
name = "argo-view"
Expand Down
Original file line number Diff line number Diff line change
@@ -1 +1 @@
# https://github.com/argoproj/argo-helm/blob/argo-workflows-0.13.1/charts/argo-workflows/values.yaml
# https://github.com/argoproj/argo-helm/blob/argo-workflows-0.22.9/charts/argo-workflows/values.yaml