Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add an env switch to both workflows that enable the new pex based workflows #23

Merged
merged 19 commits into from
Nov 10, 2022
Merged

Add an env switch to both workflows that enable the new pex based workflows #23

Show file tree
Hide file tree
Changes from 17 commits
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
31331d0
Add dagster_build_deploy_pex.yml
shalabhc Oct 28, 2022
462f4f2
Delete branch_deployments.yml
shalabhc Oct 28, 2022
f3d3060
Delete deploy.yml
shalabhc Oct 28, 2022
71a80ca
Point to the pex-testing branch of dagster-cloud-action
shalabhc Nov 1, 2022
eaf6557
Rename workflow to deploy.yml
shalabhc Nov 1, 2022
03ca53c
Pin to a specific base image
shalabhc Nov 3, 2022
3c022ca
Move env location
shalabhc Nov 3, 2022
4319d50
Pass the github token when invoking the deploy workflow
shalabhc Nov 4, 2022
4c55fba
Merge branch 'dagster-io:main' into main
shalabhc Nov 4, 2022
46fdd41
Merge in original deploy.yml
shalabhc Nov 8, 2022
8cf98c8
remove dup code
shalabhc Nov 8, 2022
84edec8
rename
shalabhc Nov 8, 2022
f7d7bbc
add branch_deployments.yml
shalabhc Nov 8, 2022
17c491b
Merge pull request #1 from dagster-io/shalabhc-merge-deploy-yml
shalabhc Nov 8, 2022
3095404
Rename code overlay to python executable
shalabhc Nov 8, 2022
2cf601f
Merge pull request #2 from dagster-io/rename-python-exe
shalabhc Nov 8, 2022
14b20c3
Change referenced tag to @pex-v0.1
shalabhc Nov 10, 2022
a741415
Fix tag references - switched the wrong one
shalabhc Nov 10, 2022
c1be31d
Checkout github.head_ref instead of github.sha for the pex based flows
shalabhc Nov 10, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
35 changes: 28 additions & 7 deletions .github/workflows/branch_deployments.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,31 +6,51 @@ concurrency:
# Cancel in-progress runs on same branch
group: ${{ github.ref }}
cancel-in-progress: true

env:
DAGSTER_CLOUD_URL: ${{ secrets.DAGSTER_CLOUD_URL }}
DAGSTER_CLOUD_API_TOKEN: ${{ secrets.DAGSTER_CLOUD_API_TOKEN }}

jobs:
parse_workspace:
dagster_cloud_default_deploy:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

rather than combining the steps into a single job, can we put the condition on the job itself and then have 3 top-level jobs?

e.g.

jobs:
  parse_workspace:
    if: env.ENABLE_PYTHON_EXECUTABLE != 'true'
    ...
  dagster_cloud_build_push:
    if: env.ENABLE_PYTHON_EXECUTABLE != 'true'
    needs: parse_workspace
    ...
  dagster_cloud_python_deploy:
    if: env.ENABLE_PYTHON_EXECUTABLE == 'true'
    steps:
      - name: Checkout
         ...
      - name: Build and deploy Python executable
        ...

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's what I tried first, but unfortunately it is not possible to access env in the job level. It is only available in steps. actions/runner#1189 (comment)

I looked for other possible ways to use a variable that can be used to switch the job, but there doesn't seem to be any.

Maybe we could create a new 'switcher' job that runs first and writes a job output which is then used in subsequent job's if, but running another job adds another 10s to spin up a new container so not sure if it is worth it.

name: Dagster Serverless Deploy
runs-on: ubuntu-latest
outputs:
build_info: ${{ steps.parse-workspace.outputs.build_info }}

steps:
- uses: actions/checkout@v3
- name: Parse cloud workspace
if: env.ENABLE_PYTHON_EXECUTABLE != 'true'
id: parse-workspace
uses: dagster-io/dagster-cloud-action/actions/utils/[email protected]
with:
dagster_cloud_file: dagster_cloud.yaml

dagster_cloud_build_push:
- name: Checkout
if: env.ENABLE_PYTHON_EXECUTABLE == 'true'
uses: actions/checkout@v3
with:
ref: ${{ github.sha }}
path: project-repo

- name: Build and deploy Python executable
if: env.ENABLE_PYTHON_EXECUTABLE == 'true'
uses: dagster-io/dagster-cloud-action/actions/[email protected]
with:
dagster_cloud_file: "$GITHUB_WORKSPACE/project-repo/dagster_cloud.yaml"
build_output_dir: "$GITHUB_WORKSPACE/build"
python_version: "3.8"
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

dagster_cloud_docker_deploy:
name: Dagster Serverless Docker Deploy
runs-on: ubuntu-latest
needs: parse_workspace
name: Dagster Serverless Deploy
if: needs.dagster_cloud_default_deploy.outputs.build_info
needs: dagster_cloud_default_deploy
strategy:
fail-fast: false
matrix:
location: ${{ fromJSON(needs.parse_workspace.outputs.build_info) }}
location: ${{ fromJSON(needs.dagster_cloud_default_deploy.outputs.build_info) }}
steps:
- name: Checkout
uses: actions/checkout@v3
Expand All @@ -46,3 +66,4 @@ jobs:
organization_id: ${{ secrets.ORGANIZATION_ID }}
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

35 changes: 28 additions & 7 deletions .github/workflows/deploy.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,26 +13,46 @@ env:
DAGSTER_CLOUD_API_TOKEN: ${{ secrets.DAGSTER_CLOUD_API_TOKEN }}

jobs:
parse_workspace:
dagster_cloud_default_deploy:
name: Dagster Serverless Deploy
runs-on: ubuntu-latest
outputs:
build_info: ${{ steps.parse-workspace.outputs.build_info }}

steps:
- uses: actions/checkout@v3
- name: Parse cloud workspace
if: env.ENABLE_PYTHON_EXECUTABLE != 'true'
id: parse-workspace
uses: dagster-io/dagster-cloud-action/actions/utils/[email protected]
uses: dagster-io/dagster-cloud-action/actions/utils/parse_workspace@pex-v0.1
with:
dagster_cloud_file: dagster_cloud.yaml

dagster_cloud_build_push:
- name: Checkout
if: env.ENABLE_PYTHON_EXECUTABLE == 'true'
uses: actions/checkout@v3
with:
ref: ${{ github.sha }}
path: project-repo

- name: Build and deploy Python executable
if: env.ENABLE_PYTHON_EXECUTABLE == 'true'
uses: dagster-io/dagster-cloud-action/actions/build_deploy_python_executable@pex-testing
with:
dagster_cloud_file: "$GITHUB_WORKSPACE/project-repo/dagster_cloud.yaml"
build_output_dir: "$GITHUB_WORKSPACE/build"
python_version: "3.8"
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

dagster_cloud_docker_deploy:
name: Dagster Serverless Docker Deploy
runs-on: ubuntu-latest
needs: parse_workspace
name: Dagster Serverless Deploy
if: needs.dagster_cloud_default_deploy.outputs.build_info
needs: dagster_cloud_default_deploy
strategy:
fail-fast: false
matrix:
location: ${{ fromJSON(needs.parse_workspace.outputs.build_info) }}
location: ${{ fromJSON(needs.dagster_cloud_default_deploy.outputs.build_info) }}
steps:
- name: Checkout
uses: actions/checkout@v3
Expand All @@ -48,3 +68,4 @@ jobs:
organization_id: ${{ secrets.ORGANIZATION_ID }}
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}