Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ci: Use uv in more places in CI #1885

Merged
merged 2 commits into from
Nov 5, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 6 additions & 4 deletions .github/workflows/check-yaml-format.yml
Original file line number Diff line number Diff line change
@@ -1,3 +1,5 @@
name: Check YAML format

on:
push:
branches:
Expand All @@ -20,10 +22,10 @@ jobs:
files: |
_data/**/*.yml

- name: Install yamllint
if: steps.changed-files.outputs.any_changed == 'true'
run: pipx install yamllint
- uses: astral-sh/setup-uv@v3
with:
version: ">=0.4.30"

- name: Lint YAML files
if: steps.changed-files.outputs.any_changed == 'true'
run: yamllint ${{ steps.changed-files.outputs.all_changed_files }}
run: uvx yamllint ${{ steps.changed-files.outputs.all_changed_files }}
10 changes: 8 additions & 2 deletions .github/workflows/metadata-extract-airbyte.yml
Original file line number Diff line number Diff line change
Expand Up @@ -22,9 +22,12 @@ jobs:

steps:
- uses: actions/[email protected]
- uses: astral-sh/setup-uv@v3
with:
version: ">=0.4.30"

- name: Install hub-utils
run: pipx install git+https://github.com/meltano/hub-utils.git@edgarrmondragon/fix/custom-python-executable
run: uv tool install git+https://github.com/meltano/hub-utils.git@main

- name: Get Variants List - Airbyte
id: get-variants-list-airbyte
Expand Down Expand Up @@ -81,9 +84,12 @@ jobs:
- uses: actions/download-artifact@v3
with:
name: extractor--${{ matrix.source-name }}--airbyte.json
- uses: astral-sh/setup-uv@v3
with:
version: ">=0.4.30"

- name: Install hub-utils
run: pipx install git+https://github.com/meltano/hub-utils.git@edgarrmondragon/fix/custom-python-executable
run: uv tool install git+https://github.com/meltano/hub-utils.git@main

- name: Upload Metadata
run: hub-utils upload-airbyte "/home/runner/work/hub/hub/_data/meltano/${{ matrix.plugin-name }}" extractor--${{ matrix.source-name }}--airbyte.json
17 changes: 14 additions & 3 deletions .github/workflows/metadata-extract-extractors.yml
Original file line number Diff line number Diff line change
Expand Up @@ -23,9 +23,12 @@ jobs:

steps:
- uses: actions/[email protected]
- uses: astral-sh/setup-uv@v3
with:
version: ">=0.4.30"

- name: Install hub-utils
run: pipx install git+https://github.com/meltano/hub-utils.git@edgarrmondragon/fix/custom-python-executable
run: uv tool install git+https://github.com/meltano/hub-utils.git@main

- name: Get Variants List - SDK Extractors (p1)
id: get-variants-list-sdk-ex-p1
Expand Down Expand Up @@ -69,8 +72,12 @@ jobs:
aws-region: us-west-2
role-session-name: "GitHubActions"

- uses: astral-sh/setup-uv@v3
with:
version: ">=0.4.30"

- name: Install hub-utils
run: pipx install git+https://github.com/meltano/hub-utils.git@edgarrmondragon/fix/custom-python-executable
run: uv tool install git+https://github.com/meltano/hub-utils.git@main

# This starts the attempts to install and extract metadata. Since we don't
# know what python versions are accepted we start with the latest and continue
Expand Down Expand Up @@ -142,8 +149,12 @@ jobs:
aws-region: us-west-2
role-session-name: "GitHubActions"

- uses: astral-sh/setup-uv@v3
with:
version: ">=0.4.30"

- name: Install hub-utils
run: pipx install git+https://github.com/meltano/hub-utils.git@edgarrmondragon/fix/custom-python-executable
run: uv tool install git+https://github.com/meltano/hub-utils.git@main

- name: Install Python
uses: actions/setup-python@v5
Expand Down
11 changes: 9 additions & 2 deletions .github/workflows/metadata-extract-loaders.yml
Original file line number Diff line number Diff line change
Expand Up @@ -22,9 +22,12 @@ jobs:

steps:
- uses: actions/[email protected]
- uses: astral-sh/setup-uv@v3
with:
version: ">=0.4.30"

- name: Install hub-utils
run: pipx install git+https://github.com/meltano/hub-utils.git@edgarrmondragon/fix/custom-python-executable
run: uv tool install git+https://github.com/meltano/hub-utils.git@main

- name: Get Variants List - SDK Loaders
id: get-variants-list-sdk-loaders
Expand Down Expand Up @@ -58,8 +61,12 @@ jobs:
aws-region: us-west-2
role-session-name: "GitHubActions"

- uses: astral-sh/setup-uv@v3
with:
version: ">=0.4.30"

- name: Install hub-utils
run: pipx install git+https://github.com/meltano/hub-utils.git@edgarrmondragon/fix/custom-python-executable
run: uv tool install git+https://github.com/meltano/hub-utils.git@main

# This starts the attempts to install and extract metadata. Since we don't
# know what python versions are accepted we start with the latest and continue
Expand Down
8 changes: 6 additions & 2 deletions .github/workflows/metadata-refresh.yml
Original file line number Diff line number Diff line change
Expand Up @@ -34,8 +34,12 @@ jobs:
aws-region: us-west-2
role-session-name: "GitHubActions"

- uses: astral-sh/setup-uv@v3
with:
version: ">=0.4.30"

- name: Install hub-utils
run: pipx install git+https://github.com/meltano/hub-utils.git@main
run: uv tool install git+https://github.com/meltano/hub-utils.git@main

- name: Download Raw JSON Output
run: hub-utils download-metadata "/home/runner/work/downloaded_data" --ignore-list-str="extractors/tap-ga4/connorflyn,extractors/tap-ga4/meltanolabs,extractors/tap-ga4/sehnem,extractors/tap-jira/meltanolabs,extractors/tap-salesforce-commerce/hotgluexyz,loaders/target-s3/crowemi,loaders/target-s3-parquet/jkausti,extractors/tap-rest-api-msdk/widen"
Expand All @@ -45,7 +49,7 @@ jobs:

- name: Get current date
id: date
run: echo "::set-output name=date::$(date +'%Y-%m-%d')"
run: echo "date=$(date +'%Y-%m-%d')" >> $GITHUB_ENV

- name: Create Pull Request
uses: peter-evans/create-pull-request@v7
Expand Down
13 changes: 9 additions & 4 deletions .github/workflows/test-plugin-command.yml
Original file line number Diff line number Diff line change
Expand Up @@ -79,20 +79,25 @@ jobs:
uses: actions/setup-python@v5
with:
# https://github.com/actions/setup-python
python-version: 3.9
python-version: 3.10
cache: 'pip'

- name: Install uv
uses: astral-sh/setup-uv@v3
with:
version: ">=0.4.30"

- name: Install plugin
id: plugin-install
run: |
pipx install ${{ env.PIP_URL }} 2>&1 | tee install-output.txt
echo "::set-output name=failed::${PIPESTATUS[0]}"
uv tool install ${{ env.PIP_URL }} 2>&1 | tee install-output.txt
echo "failed=${PIPESTATUS[0]}" >> $GITHUB_ENV

- name: Create installation failure log
if: steps.plugin-install.outputs.failed == '1'
run: |
echo "JOB_OUTPUT<<EOF" >> $GITHUB_ENV
echo "melty-bot % pipx install ${{ env.PIP_URL }}" >> $GITHUB_ENV
echo "melty-bot % uv tool install ${{ env.PIP_URL }}" >> $GITHUB_ENV
cat install-output.txt >> $GITHUB_ENV
echo "EOF" >> $GITHUB_ENV

Expand Down
2 changes: 2 additions & 0 deletions .github/workflows/test_dispatcher.yml
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ on:

jobs:
get_changed_files:
name: Get changed files
runs-on: ubuntu-latest
outputs:
matrix: ${{ steps.set-matrix.outputs.matrix }}
Expand All @@ -38,6 +39,7 @@ jobs:
run: echo "matrix={\"changed_file\":${{ steps.filter_files.outputs.filtered_files }}}" >> "$GITHUB_OUTPUT"

dispatch_tests:
name: Dispatch tests for ${{ matrix.name }} (${{ matrix.variant }})
runs-on: ubuntu-latest
needs:
- get_changed_files
Expand Down
3 changes: 2 additions & 1 deletion .github/workflows/test_meltano_add_install.yml
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,7 @@ jobs:
matrix: true

test:
name: Install plugin with Meltano
runs-on: ubuntu-latest
needs: get_changed_plugins
strategy:
Expand All @@ -45,7 +46,7 @@ jobs:
- name: Install uv
uses: astral-sh/setup-uv@v3
with:
version: "0.4.26"
version: ">=0.4.30"

- name: Install Meltano
run: uv tool install meltano
Expand Down
4 changes: 2 additions & 2 deletions _data/meltano/extractors/tap-fedidb/edgarrmondragon.yml
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ settings:
label: Batch Config Encoding Compression
name: batch_config.encoding.compression
options:
- label: Gzip
- label: GZIP
value: gzip
- label: None
value: none
Expand All @@ -37,7 +37,7 @@ settings:
label: Batch Config Encoding Format
name: batch_config.encoding.format
options:
- label: Jsonl
- label: JSONL
value: jsonl
- label: Parquet
value: parquet
Expand Down
Loading