Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Pass table migration index to WorkflowLinter #3744

Draft
wants to merge 5 commits into
base: main
Choose a base branch
from
Draft

Pass table migration index to WorkflowLinter #3744

wants to merge 5 commits into from
+18 −20

Conversation

JCZuurmond
Copy link
Member

@JCZuurmond JCZuurmond commented Feb 25, 2025 •
edited
Loading

Changes

Pass table migration index to WorkflowLinter

Linked issues

Resolves #3153

Functionality

  • modified WorkflowLinter
  • modified existing workflow: assessment

@JCZuurmond JCZuurmond added the migrate/code Abstract Syntax Trees and other dark magic label Feb 25, 2025
@JCZuurmond JCZuurmond requested a review from asnare February 25, 2025 09:05
@JCZuurmond JCZuurmond self-assigned this Feb 25, 2025
@JCZuurmond JCZuurmond requested a review from a team as a code owner February 25, 2025 09:05
@JCZuurmond
Copy link
Member Author

Running CI in https://github.com/databrickslabs/ucx/actions/runs/13517474776

@github-actions GitHub Actions
Copy link

github-actions bot commented Feb 25, 2025 •
edited
Loading

❌ 73/82 passed, 9 failed, 10 skipped, 6h0m31s total

❌ test_running_real_assessment_job: AssertionError: assert not [Row(job_id=940374825461524, job_name='[XISG] migrate-external-hiveserde-tables-in-place-experimental', task_key='migr...d5d/.XIsg/wheels/wheel-test-runner-0.56.1+120250226060617', start_line=-1, start_col=-1, end_line=-1, end_col=-1), ...] (15m51.469s) 
... (skipped 7326 bytes)
s to start...
07:51 INFO [databricks.labs.ucx.install] Installing UCX v0.56.1+720250226075123
07:51 INFO [databricks.labs.ucx.install] Creating ucx schemas...
07:51 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=scan-tables-in-mounts-experimental
07:51 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-tables-ctas
07:51 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables-in-mounts-experimental
07:51 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-data-reconciliation
07:51 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-hiveserde-tables-in-place-experimental
07:51 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups-legacy
07:51 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables
07:51 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=failing
07:51 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migration-progress-experimental
07:51 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups
07:51 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=remove-workspace-local-backup-groups
07:51 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=validate-groups-permissions
07:51 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=assessment
07:51 INFO [databricks.labs.ucx.install] Creating dashboards...
07:51 DEBUG [databricks.labs.ucx.install] Reading step folder /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/progress...
07:51 DEBUG [databricks.labs.ucx.install] Reading step folder /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/assessment...
07:51 DEBUG [databricks.labs.ucx.install] Reading step folder /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/migration...
07:51 DEBUG [databricks.labs.ucx.install] Reading step folder /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/views...
07:51 INFO [databricks.labs.ucx.install] Creating dashboard in /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/progress/main...
07:51 INFO [databricks.labs.ucx.install] Creating dashboard in /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/assessment/estimates...
07:51 INFO [databricks.labs.ucx.install] Creating dashboard in /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/assessment/main...
07:51 INFO [databricks.labs.ucx.install] Creating dashboard in /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/assessment/CLOUD_ENV...
07:51 INFO [databricks.labs.ucx.install] Creating dashboard in /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/assessment/interactive...
07:51 INFO [databricks.labs.ucx.install] Creating dashboard in /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/migration/main...
07:51 INFO [databricks.labs.ucx.install] Creating dashboard in /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/migration/groups...
07:51 INFO [databricks.labs.ucx.installer.mixins] Fetching warehouse_id from a config
07:51 INFO [databricks.labs.ucx.installer.mixins] Fetching warehouse_id from a config
07:51 INFO [databricks.labs.ucx.installer.mixins] Fetching warehouse_id from a config
07:51 INFO [databricks.labs.ucx.installer.mixins] Fetching warehouse_id from a config
07:51 INFO [databricks.labs.ucx.installer.mixins] Fetching warehouse_id from a config
07:51 INFO [databricks.labs.ucx.installer.mixins] Fetching warehouse_id from a config
07:51 INFO [databricks.labs.ucx.installer.mixins] Fetching warehouse_id from a config
07:51 INFO [databricks.labs.ucx.install] Installation completed successfully! Please refer to the https://DATABRICKS_HOST/#workspace/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.Q0ZI/README for the next steps.
07:51 DEBUG [databricks.labs.ucx.installer.workflows] starting assessment job: https://DATABRICKS_HOST#job/331973257922069
07:51 INFO [databricks.labs.ucx.installer.workflows] Started assessment job: https://DATABRICKS_HOST#job/331973257922069/runs/803498728152131
07:51 DEBUG [databricks.labs.ucx.installer.workflows] Validating assessment workflow: https://DATABRICKS_HOST#job/331973257922069
07:51 INFO [databricks.labs.ucx.installer.workflows] Identified a run in progress waiting for run completion
07:59 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.dummy_snojr.tables] fetching tables inventory
07:59 INFO [databricks.labs.ucx.install] Deleting UCX v0.56.1+720250226075123 from https://DATABRICKS_HOST
07:59 INFO [databricks.labs.ucx.install] Deleting inventory database dummy_snojr
07:59 INFO [databricks.labs.ucx.installer.workflows] Removing job_id=683022803499845, as it is no longer needed
07:59 INFO [databricks.labs.ucx.installer.workflows] Removing job_id=794041170790193, as it is no longer needed
07:59 INFO [databricks.labs.ucx.installer.workflows] Removing job_id=402070617677768, as it is no longer needed
07:59 INFO [databricks.labs.ucx.installer.workflows] Removing job_id=867460535215029, as it is no longer needed
07:59 INFO [databricks.labs.ucx.installer.workflows] Removing job_id=187432714352459, as it is no longer needed
07:59 INFO [databricks.labs.ucx.installer.workflows] Removing job_id=503175453470855, as it is no longer needed
07:59 INFO [databricks.labs.ucx.installer.workflows] Removing job_id=285859525999425, as it is no longer needed
07:59 INFO [databricks.labs.ucx.installer.workflows] Removing job_id=775701684762359, as it is no longer needed
07:59 INFO [databricks.labs.ucx.installer.workflows] Removing job_id=756875191405496, as it is no longer needed
08:00 INFO [databricks.labs.ucx.installer.workflows] Removing job_id=754479577773387, as it is no longer needed
08:00 INFO [databricks.labs.ucx.installer.workflows] Removing job_id=139858237993484, as it is no longer needed
08:00 INFO [databricks.labs.ucx.installer.workflows] Removing job_id=460321927634560, as it is no longer needed
08:00 INFO [databricks.labs.ucx.installer.workflows] Removing job_id=331973257922069, as it is no longer needed
08:00 INFO [databricks.labs.ucx.install] Deleting cluster policy
08:00 INFO [databricks.labs.ucx.install] Deleting secret scope
08:00 INFO [databricks.labs.ucx.install] UnInstalling UCX complete
[gw7] linux -- Python 3.10.16 /home/runner/work/ucx/ucx/.venv/bin/python
❌ test_hiveserde_table_in_place_migration_job[migrate-external-hiveserde-tables-in-place-experimental]: AssertionError: Workflow failed: assessment (10m16.395s) 
... (skipped 7103 bytes)
est] Waiting for clusters to start...
08:06 INFO [databricks.labs.ucx.install] Installing UCX v0.56.1+720250226080620
08:06 INFO [databricks.labs.ucx.install] Creating ucx schemas...
08:06 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-tables-ctas
08:06 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables-in-mounts-experimental
08:06 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups
08:06 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migration-progress-experimental
08:06 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-data-reconciliation
08:06 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=scan-tables-in-mounts-experimental
08:06 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=validate-groups-permissions
08:06 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables
08:06 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=remove-workspace-local-backup-groups
08:06 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=assessment
08:06 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=failing
08:06 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups-legacy
08:06 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-hiveserde-tables-in-place-experimental
08:06 INFO [databricks.labs.ucx.install] Creating dashboards...
08:06 DEBUG [databricks.labs.ucx.install] Reading step folder /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/progress...
08:06 DEBUG [databricks.labs.ucx.install] Reading step folder /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/assessment...
08:06 DEBUG [databricks.labs.ucx.install] Reading step folder /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/migration...
08:06 DEBUG [databricks.labs.ucx.install] Reading step folder /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/views...
08:06 INFO [databricks.labs.ucx.install] Creating dashboard in /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/progress/main...
08:06 INFO [databricks.labs.ucx.install] Creating dashboard in /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/assessment/estimates...
08:06 INFO [databricks.labs.ucx.install] Creating dashboard in /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/assessment/main...
08:06 INFO [databricks.labs.ucx.install] Creating dashboard in /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/assessment/CLOUD_ENV...
08:06 INFO [databricks.labs.ucx.install] Creating dashboard in /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/assessment/interactive...
08:06 INFO [databricks.labs.ucx.install] Creating dashboard in /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/migration/main...
08:06 INFO [databricks.labs.ucx.install] Creating dashboard in /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/migration/groups...
08:06 INFO [databricks.labs.ucx.installer.mixins] Fetching warehouse_id from a config
08:06 INFO [databricks.labs.ucx.installer.mixins] Fetching warehouse_id from a config
08:06 INFO [databricks.labs.ucx.installer.mixins] Fetching warehouse_id from a config
08:06 INFO [databricks.labs.ucx.installer.mixins] Fetching warehouse_id from a config
08:06 INFO [databricks.labs.ucx.installer.mixins] Fetching warehouse_id from a config
08:06 INFO [databricks.labs.ucx.installer.mixins] Fetching warehouse_id from a config
08:06 INFO [databricks.labs.ucx.installer.mixins] Fetching warehouse_id from a config
08:06 INFO [databricks.labs.ucx.install] Installation completed successfully! Please refer to the https://DATABRICKS_HOST/#workspace/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.2mY3/README for the next steps.
08:07 INFO [databricks.labs.ucx.progress.install] Installation completed successfully!
08:07 DEBUG [databricks.labs.ucx.installer.workflows] starting assessment job: https://DATABRICKS_HOST#job/1025756626044127
08:07 INFO [databricks.labs.ucx.installer.workflows] Started assessment job: https://DATABRICKS_HOST#job/1025756626044127/runs/695380224532996
08:07 DEBUG [databricks.labs.ucx.installer.workflows] Validating assessment workflow: https://DATABRICKS_HOST#job/1025756626044127
08:07 INFO [databricks.labs.ucx.installer.workflows] Identified a run in progress waiting for run completion
08:14 INFO [databricks.labs.ucx.install] Deleting UCX v0.56.1+720250226080620 from https://DATABRICKS_HOST
08:14 INFO [databricks.labs.ucx.install] Deleting inventory database dummy_sj9c7
08:14 INFO [databricks.labs.ucx.installer.workflows] Removing job_id=454879326927107, as it is no longer needed
08:14 INFO [databricks.labs.ucx.installer.workflows] Removing job_id=522467118886249, as it is no longer needed
08:14 INFO [databricks.labs.ucx.installer.workflows] Removing job_id=243835687546273, as it is no longer needed
08:14 INFO [databricks.labs.ucx.installer.workflows] Removing job_id=442816321513389, as it is no longer needed
08:14 INFO [databricks.labs.ucx.installer.workflows] Removing job_id=298102505916411, as it is no longer needed
08:14 INFO [databricks.labs.ucx.installer.workflows] Removing job_id=601306153330423, as it is no longer needed
08:14 INFO [databricks.labs.ucx.installer.workflows] Removing job_id=852798984081214, as it is no longer needed
08:14 INFO [databricks.labs.ucx.installer.workflows] Removing job_id=303919107061306, as it is no longer needed
08:14 INFO [databricks.labs.ucx.installer.workflows] Removing job_id=606028570411758, as it is no longer needed
08:14 INFO [databricks.labs.ucx.installer.workflows] Removing job_id=1025756626044127, as it is no longer needed
08:14 INFO [databricks.labs.ucx.installer.workflows] Removing job_id=286393923869448, as it is no longer needed
08:14 INFO [databricks.labs.ucx.installer.workflows] Removing job_id=100686519718524, as it is no longer needed
08:14 INFO [databricks.labs.ucx.installer.workflows] Removing job_id=618190473490400, as it is no longer needed
08:14 INFO [databricks.labs.ucx.install] Deleting cluster policy
08:14 INFO [databricks.labs.ucx.install] Deleting secret scope
08:14 INFO [databricks.labs.ucx.install] UnInstalling UCX complete
[gw4] linux -- Python 3.10.16 /home/runner/work/ucx/ucx/.venv/bin/python
❌ test_table_migration_job_refreshes_migration_status[hiveserde-migrate-external-hiveserde-tables-in-place-experimental]: AssertionError: Workflow failed: assessment (7m27.059s) 
... (skipped 6736 bytes)
 Waiting for clusters to start...
08:08 INFO [databricks.labs.ucx.install] Installing UCX v0.56.1+720250226080830
08:08 INFO [databricks.labs.ucx.install] Creating ucx schemas...
08:08 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=remove-workspace-local-backup-groups
08:08 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migration-progress-experimental
08:08 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups-legacy
08:08 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=validate-groups-permissions
08:08 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-tables-ctas
08:08 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-hiveserde-tables-in-place-experimental
08:08 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=scan-tables-in-mounts-experimental
08:08 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups
08:08 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=failing
08:08 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables-in-mounts-experimental
08:08 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=assessment
08:08 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-data-reconciliation
08:08 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables
08:08 INFO [databricks.labs.ucx.install] Creating dashboards...
08:08 DEBUG [databricks.labs.ucx.install] Reading step folder /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/progress...
08:08 DEBUG [databricks.labs.ucx.install] Reading step folder /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/assessment...
08:08 DEBUG [databricks.labs.ucx.install] Reading step folder /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/migration...
08:08 DEBUG [databricks.labs.ucx.install] Reading step folder /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/views...
08:08 INFO [databricks.labs.ucx.install] Creating dashboard in /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/progress/main...
08:08 INFO [databricks.labs.ucx.install] Creating dashboard in /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/assessment/estimates...
08:08 INFO [databricks.labs.ucx.install] Creating dashboard in /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/assessment/main...
08:08 INFO [databricks.labs.ucx.install] Creating dashboard in /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/assessment/CLOUD_ENV...
08:08 INFO [databricks.labs.ucx.install] Creating dashboard in /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/assessment/interactive...
08:08 INFO [databricks.labs.ucx.install] Creating dashboard in /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/migration/main...
08:08 INFO [databricks.labs.ucx.install] Creating dashboard in /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/migration/groups...
08:08 INFO [databricks.labs.ucx.installer.mixins] Fetching warehouse_id from a config
08:08 INFO [databricks.labs.ucx.installer.mixins] Fetching warehouse_id from a config
08:08 INFO [databricks.labs.ucx.installer.mixins] Fetching warehouse_id from a config
08:08 INFO [databricks.labs.ucx.installer.mixins] Fetching warehouse_id from a config
08:08 INFO [databricks.labs.ucx.installer.mixins] Fetching warehouse_id from a config
08:08 INFO [databricks.labs.ucx.installer.mixins] Fetching warehouse_id from a config
08:08 INFO [databricks.labs.ucx.installer.mixins] Fetching warehouse_id from a config
08:08 INFO [databricks.labs.ucx.install] Installation completed successfully! Please refer to the https://DATABRICKS_HOST/#workspace/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.RPon/README for the next steps.
08:08 INFO [databricks.labs.ucx.progress.install] Installation completed successfully!
08:08 DEBUG [databricks.labs.ucx.installer.workflows] starting assessment job: https://DATABRICKS_HOST#job/1060611191279858
08:08 INFO [databricks.labs.ucx.installer.workflows] Started assessment job: https://DATABRICKS_HOST#job/1060611191279858/runs/812886628758431
08:08 DEBUG [databricks.labs.ucx.installer.workflows] Validating assessment workflow: https://DATABRICKS_HOST#job/1060611191279858
08:08 INFO [databricks.labs.ucx.installer.workflows] Identified a run in progress waiting for run completion
08:14 INFO [databricks.labs.ucx.install] Deleting UCX v0.56.1+720250226080830 from https://DATABRICKS_HOST
08:14 INFO [databricks.labs.ucx.install] Deleting inventory database dummy_scuh6
08:15 INFO [databricks.labs.ucx.installer.workflows] Removing job_id=654173588312283, as it is no longer needed
08:15 INFO [databricks.labs.ucx.installer.workflows] Removing job_id=753665197041477, as it is no longer needed
08:15 INFO [databricks.labs.ucx.installer.workflows] Removing job_id=1025092694020848, as it is no longer needed
08:15 INFO [databricks.labs.ucx.installer.workflows] Removing job_id=1123760544867754, as it is no longer needed
08:15 INFO [databricks.labs.ucx.installer.workflows] Removing job_id=134919207851083, as it is no longer needed
08:15 INFO [databricks.labs.ucx.installer.workflows] Removing job_id=343879823782438, as it is no longer needed
08:15 INFO [databricks.labs.ucx.installer.workflows] Removing job_id=838245179207988, as it is no longer needed
08:15 INFO [databricks.labs.ucx.installer.workflows] Removing job_id=1105962601652353, as it is no longer needed
08:15 INFO [databricks.labs.ucx.installer.workflows] Removing job_id=292688352474674, as it is no longer needed
08:15 INFO [databricks.labs.ucx.installer.workflows] Removing job_id=411916144346778, as it is no longer needed
08:15 INFO [databricks.labs.ucx.installer.workflows] Removing job_id=1060611191279858, as it is no longer needed
08:15 INFO [databricks.labs.ucx.installer.workflows] Removing job_id=486374330130201, as it is no longer needed
08:15 INFO [databricks.labs.ucx.installer.workflows] Removing job_id=1012387448005778, as it is no longer needed
08:15 INFO [databricks.labs.ucx.install] Deleting cluster policy
08:15 INFO [databricks.labs.ucx.install] Deleting secret scope
08:15 INFO [databricks.labs.ucx.install] UnInstalling UCX complete
[gw9] linux -- Python 3.10.16 /home/runner/work/ucx/ucx/.venv/bin/python
❌ test_table_migration_job_refreshes_migration_status[hiveserde-migrate-external-tables-ctas]: AssertionError: Workflow failed: assessment (10m35.072s) 
... (skipped 6731 bytes)
est] Waiting for clusters to start...
08:05 INFO [databricks.labs.ucx.install] Installing UCX v0.56.1+720250226080524
08:05 INFO [databricks.labs.ucx.install] Creating ucx schemas...
08:05 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=remove-workspace-local-backup-groups
08:05 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=assessment
08:05 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=scan-tables-in-mounts-experimental
08:05 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups-legacy
08:05 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-hiveserde-tables-in-place-experimental
08:05 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-tables-ctas
08:05 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables-in-mounts-experimental
08:05 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-data-reconciliation
08:05 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migration-progress-experimental
08:05 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=validate-groups-permissions
08:05 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups
08:05 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=failing
08:05 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables
08:05 INFO [databricks.labs.ucx.install] Creating dashboards...
08:05 DEBUG [databricks.labs.ucx.install] Reading step folder /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/progress...
08:05 DEBUG [databricks.labs.ucx.install] Reading step folder /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/assessment...
08:05 DEBUG [databricks.labs.ucx.install] Reading step folder /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/migration...
08:05 DEBUG [databricks.labs.ucx.install] Reading step folder /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/views...
08:05 INFO [databricks.labs.ucx.install] Creating dashboard in /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/progress/main...
08:05 INFO [databricks.labs.ucx.install] Creating dashboard in /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/assessment/estimates...
08:05 INFO [databricks.labs.ucx.install] Creating dashboard in /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/assessment/main...
08:05 INFO [databricks.labs.ucx.install] Creating dashboard in /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/assessment/CLOUD_ENV...
08:05 INFO [databricks.labs.ucx.install] Creating dashboard in /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/assessment/interactive...
08:05 INFO [databricks.labs.ucx.install] Creating dashboard in /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/migration/main...
08:05 INFO [databricks.labs.ucx.install] Creating dashboard in /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/migration/groups...
08:05 INFO [databricks.labs.ucx.installer.mixins] Fetching warehouse_id from a config
08:05 INFO [databricks.labs.ucx.installer.mixins] Fetching warehouse_id from a config
08:05 INFO [databricks.labs.ucx.installer.mixins] Fetching warehouse_id from a config
08:05 INFO [databricks.labs.ucx.installer.mixins] Fetching warehouse_id from a config
08:05 INFO [databricks.labs.ucx.installer.mixins] Fetching warehouse_id from a config
08:05 INFO [databricks.labs.ucx.installer.mixins] Fetching warehouse_id from a config
08:05 INFO [databricks.labs.ucx.installer.mixins] Fetching warehouse_id from a config
08:05 INFO [databricks.labs.ucx.install] Installation completed successfully! Please refer to the https://DATABRICKS_HOST/#workspace/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.9vgn/README for the next steps.
08:05 INFO [databricks.labs.ucx.progress.install] Installation completed successfully!
08:05 DEBUG [databricks.labs.ucx.installer.workflows] starting assessment job: https://DATABRICKS_HOST#job/1116862638198294
08:05 INFO [databricks.labs.ucx.installer.workflows] Started assessment job: https://DATABRICKS_HOST#job/1116862638198294/runs/22322554357196
08:05 DEBUG [databricks.labs.ucx.installer.workflows] Validating assessment workflow: https://DATABRICKS_HOST#job/1116862638198294
08:05 INFO [databricks.labs.ucx.installer.workflows] Identified a run in progress waiting for run completion
08:14 INFO [databricks.labs.ucx.install] Deleting UCX v0.56.1+720250226080524 from https://DATABRICKS_HOST
08:14 INFO [databricks.labs.ucx.install] Deleting inventory database dummy_srbsi
08:14 INFO [databricks.labs.ucx.installer.workflows] Removing job_id=820770360712812, as it is no longer needed
08:14 INFO [databricks.labs.ucx.installer.workflows] Removing job_id=1116862638198294, as it is no longer needed
08:14 INFO [databricks.labs.ucx.installer.workflows] Removing job_id=216958261575230, as it is no longer needed
08:14 INFO [databricks.labs.ucx.installer.workflows] Removing job_id=206773275654510, as it is no longer needed
08:14 INFO [databricks.labs.ucx.installer.workflows] Removing job_id=459680937100178, as it is no longer needed
08:14 INFO [databricks.labs.ucx.installer.workflows] Removing job_id=894864139278413, as it is no longer needed
08:14 INFO [databricks.labs.ucx.installer.workflows] Removing job_id=1052171006992010, as it is no longer needed
08:14 INFO [databricks.labs.ucx.installer.workflows] Removing job_id=799738870800858, as it is no longer needed
08:14 INFO [databricks.labs.ucx.installer.workflows] Removing job_id=752127156047935, as it is no longer needed
08:14 INFO [databricks.labs.ucx.installer.workflows] Removing job_id=517885828865558, as it is no longer needed
08:14 INFO [databricks.labs.ucx.installer.workflows] Removing job_id=841082455075445, as it is no longer needed
08:14 INFO [databricks.labs.ucx.installer.workflows] Removing job_id=163082355633509, as it is no longer needed
08:14 INFO [databricks.labs.ucx.installer.workflows] Removing job_id=178627831763396, as it is no longer needed
08:15 INFO [databricks.labs.ucx.install] Deleting cluster policy
08:15 INFO [databricks.labs.ucx.install] Deleting secret scope
08:15 INFO [databricks.labs.ucx.install] UnInstalling UCX complete
[gw5] linux -- Python 3.10.16 /home/runner/work/ucx/ucx/.venv/bin/python
❌ test_hiveserde_table_in_place_migration_job[migrate-external-tables-ctas]: AssertionError: Workflow failed: assessment (8m0.18s) 
... (skipped 7097 bytes)
nftest] Waiting for clusters to start...
08:08 INFO [databricks.labs.ucx.install] Installing UCX v0.56.1+720250226080811
08:08 INFO [databricks.labs.ucx.install] Creating ucx schemas...
08:08 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migration-progress-experimental
08:08 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables
08:08 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=assessment
08:08 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=remove-workspace-local-backup-groups
08:08 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=validate-groups-permissions
08:08 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=scan-tables-in-mounts-experimental
08:08 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-data-reconciliation
08:08 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-hiveserde-tables-in-place-experimental
08:08 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-tables-ctas
08:08 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables-in-mounts-experimental
08:08 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups-legacy
08:08 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=failing
08:08 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups
08:08 INFO [databricks.labs.ucx.install] Creating dashboards...
08:08 DEBUG [databricks.labs.ucx.install] Reading step folder /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/progress...
08:08 DEBUG [databricks.labs.ucx.install] Reading step folder /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/assessment...
08:08 DEBUG [databricks.labs.ucx.install] Reading step folder /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/migration...
08:08 DEBUG [databricks.labs.ucx.install] Reading step folder /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/views...
08:08 INFO [databricks.labs.ucx.install] Creating dashboard in /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/progress/main...
08:08 INFO [databricks.labs.ucx.install] Creating dashboard in /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/assessment/estimates...
08:08 INFO [databricks.labs.ucx.install] Creating dashboard in /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/assessment/main...
08:08 INFO [databricks.labs.ucx.install] Creating dashboard in /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/assessment/CLOUD_ENV...
08:08 INFO [databricks.labs.ucx.install] Creating dashboard in /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/assessment/interactive...
08:08 INFO [databricks.labs.ucx.install] Creating dashboard in /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/migration/main...
08:08 INFO [databricks.labs.ucx.install] Creating dashboard in /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/migration/groups...
08:08 INFO [databricks.labs.ucx.installer.mixins] Fetching warehouse_id from a config
08:08 INFO [databricks.labs.ucx.installer.mixins] Fetching warehouse_id from a config
08:08 INFO [databricks.labs.ucx.installer.mixins] Fetching warehouse_id from a config
08:08 INFO [databricks.labs.ucx.installer.mixins] Fetching warehouse_id from a config
08:08 INFO [databricks.labs.ucx.installer.mixins] Fetching warehouse_id from a config
08:08 INFO [databricks.labs.ucx.installer.mixins] Fetching warehouse_id from a config
08:08 INFO [databricks.labs.ucx.installer.mixins] Fetching warehouse_id from a config
08:08 INFO [databricks.labs.ucx.install] Installation completed successfully! Please refer to the https://DATABRICKS_HOST/#workspace/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.0pIv/README for the next steps.
08:08 INFO [databricks.labs.ucx.progress.install] Installation completed successfully!
08:08 DEBUG [databricks.labs.ucx.installer.workflows] starting assessment job: https://DATABRICKS_HOST#job/842337714904004
08:08 INFO [databricks.labs.ucx.installer.workflows] Started assessment job: https://DATABRICKS_HOST#job/842337714904004/runs/743219148437101
08:08 DEBUG [databricks.labs.ucx.installer.workflows] Validating assessment workflow: https://DATABRICKS_HOST#job/842337714904004
08:08 INFO [databricks.labs.ucx.installer.workflows] Identified a run in progress waiting for run completion
08:14 INFO [databricks.labs.ucx.install] Deleting UCX v0.56.1+720250226080811 from https://DATABRICKS_HOST
08:14 INFO [databricks.labs.ucx.install] Deleting inventory database dummy_smct9
08:15 INFO [databricks.labs.ucx.installer.workflows] Removing job_id=51096055549167, as it is no longer needed
08:15 INFO [databricks.labs.ucx.installer.workflows] Removing job_id=1042073722152969, as it is no longer needed
08:15 INFO [databricks.labs.ucx.installer.workflows] Removing job_id=842337714904004, as it is no longer needed
08:15 INFO [databricks.labs.ucx.installer.workflows] Removing job_id=1123505862177049, as it is no longer needed
08:15 INFO [databricks.labs.ucx.installer.workflows] Removing job_id=900817730537287, as it is no longer needed
08:15 INFO [databricks.labs.ucx.installer.workflows] Removing job_id=370696822566734, as it is no longer needed
08:15 INFO [databricks.labs.ucx.installer.workflows] Removing job_id=20413038447774, as it is no longer needed
08:15 INFO [databricks.labs.ucx.installer.workflows] Removing job_id=659218328500115, as it is no longer needed
08:15 INFO [databricks.labs.ucx.installer.workflows] Removing job_id=916671578398283, as it is no longer needed
08:15 INFO [databricks.labs.ucx.installer.workflows] Removing job_id=1005833567572898, as it is no longer needed
08:15 INFO [databricks.labs.ucx.installer.workflows] Removing job_id=850658303831007, as it is no longer needed
08:15 INFO [databricks.labs.ucx.installer.workflows] Removing job_id=866164379426785, as it is no longer needed
08:15 INFO [databricks.labs.ucx.installer.workflows] Removing job_id=336477638666203, as it is no longer needed
08:15 INFO [databricks.labs.ucx.install] Deleting cluster policy
08:15 INFO [databricks.labs.ucx.install] Deleting secret scope
08:15 INFO [databricks.labs.ucx.install] UnInstalling UCX complete
[gw1] linux -- Python 3.10.16 /home/runner/work/ucx/ucx/.venv/bin/python
❌ test_table_migration_job_publishes_remaining_tables: AssertionError: Workflow failed: assessment (7m29.675s) 
... (skipped 7094 bytes)
.conftest] Waiting for clusters to start...
08:08 INFO [databricks.labs.ucx.install] Installing UCX v0.56.1+720250226080842
08:08 INFO [databricks.labs.ucx.install] Creating ucx schemas...
08:08 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-tables-ctas
08:08 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=validate-groups-permissions
08:08 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups
08:08 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-hiveserde-tables-in-place-experimental
08:08 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=remove-workspace-local-backup-groups
08:08 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migration-progress-experimental
08:08 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=assessment
08:08 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=scan-tables-in-mounts-experimental
08:08 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables-in-mounts-experimental
08:08 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=failing
08:08 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables
08:08 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups-legacy
08:08 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-data-reconciliation
08:08 INFO [databricks.labs.ucx.install] Creating dashboards...
08:08 DEBUG [databricks.labs.ucx.install] Reading step folder /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/progress...
08:08 DEBUG [databricks.labs.ucx.install] Reading step folder /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/assessment...
08:08 DEBUG [databricks.labs.ucx.install] Reading step folder /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/migration...
08:08 DEBUG [databricks.labs.ucx.install] Reading step folder /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/views...
08:08 INFO [databricks.labs.ucx.install] Creating dashboard in /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/progress/main...
08:08 INFO [databricks.labs.ucx.install] Creating dashboard in /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/assessment/estimates...
08:08 INFO [databricks.labs.ucx.install] Creating dashboard in /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/assessment/main...
08:08 INFO [databricks.labs.ucx.install] Creating dashboard in /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/assessment/CLOUD_ENV...
08:08 INFO [databricks.labs.ucx.install] Creating dashboard in /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/assessment/interactive...
08:08 INFO [databricks.labs.ucx.install] Creating dashboard in /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/migration/main...
08:08 INFO [databricks.labs.ucx.install] Creating dashboard in /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/migration/groups...
08:08 INFO [databricks.labs.ucx.installer.mixins] Fetching warehouse_id from a config
08:08 INFO [databricks.labs.ucx.installer.mixins] Fetching warehouse_id from a config
08:08 INFO [databricks.labs.ucx.installer.mixins] Fetching warehouse_id from a config
08:08 INFO [databricks.labs.ucx.installer.mixins] Fetching warehouse_id from a config
08:08 INFO [databricks.labs.ucx.installer.mixins] Fetching warehouse_id from a config
08:08 INFO [databricks.labs.ucx.installer.mixins] Fetching warehouse_id from a config
08:08 INFO [databricks.labs.ucx.installer.mixins] Fetching warehouse_id from a config
08:08 INFO [databricks.labs.ucx.install] Installation completed successfully! Please refer to the https://DATABRICKS_HOST/#workspace/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.qUMD/README for the next steps.
08:09 INFO [databricks.labs.ucx.progress.install] Installation completed successfully!
08:09 DEBUG [databricks.labs.ucx.installer.workflows] starting assessment job: https://DATABRICKS_HOST#job/885270298283852
08:09 INFO [databricks.labs.ucx.installer.workflows] Started assessment job: https://DATABRICKS_HOST#job/885270298283852/runs/851124350544596
08:09 DEBUG [databricks.labs.ucx.installer.workflows] Validating assessment workflow: https://DATABRICKS_HOST#job/885270298283852
08:09 INFO [databricks.labs.ucx.installer.workflows] Identified a run in progress waiting for run completion
08:14 INFO [databricks.labs.ucx.install] Deleting UCX v0.56.1+720250226080842 from https://DATABRICKS_HOST
08:14 INFO [databricks.labs.ucx.install] Deleting inventory database dummy_syeb4
08:14 INFO [databricks.labs.ucx.installer.workflows] Removing job_id=134265372149512, as it is no longer needed
08:14 INFO [databricks.labs.ucx.installer.workflows] Removing job_id=83364861828069, as it is no longer needed
08:14 INFO [databricks.labs.ucx.installer.workflows] Removing job_id=558015135574848, as it is no longer needed
08:14 INFO [databricks.labs.ucx.installer.workflows] Removing job_id=360614870447201, as it is no longer needed
08:14 INFO [databricks.labs.ucx.installer.workflows] Removing job_id=154700735250957, as it is no longer needed
08:14 INFO [databricks.labs.ucx.installer.workflows] Removing job_id=201331978733663, as it is no longer needed
08:14 INFO [databricks.labs.ucx.installer.workflows] Removing job_id=885270298283852, as it is no longer needed
08:14 INFO [databricks.labs.ucx.installer.workflows] Removing job_id=713082854120540, as it is no longer needed
08:14 INFO [databricks.labs.ucx.installer.workflows] Removing job_id=651927627970736, as it is no longer needed
08:14 INFO [databricks.labs.ucx.installer.workflows] Removing job_id=621084752380295, as it is no longer needed
08:14 INFO [databricks.labs.ucx.installer.workflows] Removing job_id=87490735524596, as it is no longer needed
08:14 INFO [databricks.labs.ucx.installer.workflows] Removing job_id=410544680912073, as it is no longer needed
08:14 INFO [databricks.labs.ucx.installer.workflows] Removing job_id=224222314091362, as it is no longer needed
08:14 INFO [databricks.labs.ucx.install] Deleting cluster policy
08:14 INFO [databricks.labs.ucx.install] Deleting secret scope
08:14 INFO [databricks.labs.ucx.install] UnInstalling UCX complete
[gw6] linux -- Python 3.10.16 /home/runner/work/ucx/ucx/.venv/bin/python
❌ test_table_migration_job_refreshes_migration_status[regular-migrate-tables]: databricks.sdk.errors.sdk.OperationFailed: failed to reach TERMINATED or SKIPPED, got RunLifeCycleState.INTERNAL_ERROR: Task crawl_mounts failed with message: Could not reach driver of cluster DATABRICKS_CLUSTER_ID. (19m36.333s) 
... (skipped 6885 bytes)
nftest] Waiting for clusters to start...
08:17 INFO [databricks.labs.ucx.install] Installing UCX v0.56.1+720250226081707
08:17 INFO [databricks.labs.ucx.install] Creating ucx schemas...
08:17 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=remove-workspace-local-backup-groups
08:17 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups
08:17 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-hiveserde-tables-in-place-experimental
08:17 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=validate-groups-permissions
08:17 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables-in-mounts-experimental
08:17 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups-legacy
08:17 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-tables-ctas
08:17 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=assessment
08:17 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-data-reconciliation
08:17 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=failing
08:17 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=scan-tables-in-mounts-experimental
08:17 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migration-progress-experimental
08:17 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables
08:17 INFO [databricks.labs.ucx.install] Creating dashboards...
08:17 DEBUG [databricks.labs.ucx.install] Reading step folder /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/progress...
08:17 DEBUG [databricks.labs.ucx.install] Reading step folder /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/assessment...
08:17 DEBUG [databricks.labs.ucx.install] Reading step folder /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/migration...
08:17 DEBUG [databricks.labs.ucx.install] Reading step folder /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/views...
08:17 INFO [databricks.labs.ucx.install] Creating dashboard in /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/progress/main...
08:17 INFO [databricks.labs.ucx.install] Creating dashboard in /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/assessment/estimates...
08:17 INFO [databricks.labs.ucx.install] Creating dashboard in /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/assessment/main...
08:17 INFO [databricks.labs.ucx.install] Creating dashboard in /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/assessment/CLOUD_ENV...
08:17 INFO [databricks.labs.ucx.install] Creating dashboard in /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/assessment/interactive...
08:17 INFO [databricks.labs.ucx.install] Creating dashboard in /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/migration/main...
08:17 INFO [databricks.labs.ucx.install] Creating dashboard in /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/migration/groups...
08:17 INFO [databricks.labs.ucx.installer.mixins] Fetching warehouse_id from a config
08:17 INFO [databricks.labs.ucx.installer.mixins] Fetching warehouse_id from a config
08:17 INFO [databricks.labs.ucx.installer.mixins] Fetching warehouse_id from a config
08:17 INFO [databricks.labs.ucx.installer.mixins] Fetching warehouse_id from a config
08:17 INFO [databricks.labs.ucx.installer.mixins] Fetching warehouse_id from a config
08:17 INFO [databricks.labs.ucx.installer.mixins] Fetching warehouse_id from a config
08:17 INFO [databricks.labs.ucx.installer.mixins] Fetching warehouse_id from a config
08:17 INFO [databricks.labs.ucx.install] Installation completed successfully! Please refer to the https://DATABRICKS_HOST/#workspace/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.kEEq/README for the next steps.
08:17 INFO [databricks.labs.ucx.progress.install] Installation completed successfully!
08:17 DEBUG [databricks.labs.ucx.installer.workflows] starting assessment job: https://DATABRICKS_HOST#job/336769684000453
08:17 INFO [databricks.labs.ucx.installer.workflows] Started assessment job: https://DATABRICKS_HOST#job/336769684000453/runs/665777791607637
08:17 DEBUG [databricks.labs.ucx.installer.workflows] Validating assessment workflow: https://DATABRICKS_HOST#job/336769684000453
08:17 INFO [databricks.labs.ucx.installer.workflows] Identified a run in progress waiting for run completion
08:34 INFO [databricks.labs.ucx.install] Deleting UCX v0.56.1+720250226081707 from https://DATABRICKS_HOST
08:34 INFO [databricks.labs.ucx.install] Deleting inventory database dummy_sqhur
08:34 INFO [databricks.labs.ucx.installer.workflows] Removing job_id=997207009936667, as it is no longer needed
08:34 INFO [databricks.labs.ucx.installer.workflows] Removing job_id=760846191109069, as it is no longer needed
08:34 INFO [databricks.labs.ucx.installer.workflows] Removing job_id=1094788742742492, as it is no longer needed
08:34 INFO [databricks.labs.ucx.installer.workflows] Removing job_id=181075172489876, as it is no longer needed
08:34 INFO [databricks.labs.ucx.installer.workflows] Removing job_id=887574172455199, as it is no longer needed
08:34 INFO [databricks.labs.ucx.installer.workflows] Removing job_id=964762938639726, as it is no longer needed
08:34 INFO [databricks.labs.ucx.installer.workflows] Removing job_id=954452061456195, as it is no longer needed
08:34 INFO [databricks.labs.ucx.installer.workflows] Removing job_id=336769684000453, as it is no longer needed
08:34 INFO [databricks.labs.ucx.installer.workflows] Removing job_id=990307984994541, as it is no longer needed
08:34 INFO [databricks.labs.ucx.installer.workflows] Removing job_id=390633267347691, as it is no longer needed
08:34 INFO [databricks.labs.ucx.installer.workflows] Removing job_id=578625724752110, as it is no longer needed
08:34 INFO [databricks.labs.ucx.installer.workflows] Removing job_id=867584773947436, as it is no longer needed
08:34 INFO [databricks.labs.ucx.installer.workflows] Removing job_id=743671807921674, as it is no longer needed
08:34 INFO [databricks.labs.ucx.install] Deleting cluster policy
08:34 INFO [databricks.labs.ucx.install] Deleting secret scope
08:34 INFO [databricks.labs.ucx.install] UnInstalling UCX complete
[gw3] linux -- Python 3.10.16 /home/runner/work/ucx/ucx/.venv/bin/python
❌ test_hiveserde_table_ctas_migration_job: AssertionError: Workflow failed: assessment (21m27.921s) 
... (skipped 7097 bytes)
nftest] Waiting for clusters to start...
08:15 INFO [databricks.labs.ucx.install] Installing UCX v0.56.1+720250226081557
08:15 INFO [databricks.labs.ucx.install] Creating ucx schemas...
08:16 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-tables-ctas
08:16 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables-in-mounts-experimental
08:16 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups
08:16 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migration-progress-experimental
08:16 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-data-reconciliation
08:16 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=scan-tables-in-mounts-experimental
08:16 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=validate-groups-permissions
08:16 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables
08:16 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=remove-workspace-local-backup-groups
08:16 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=assessment
08:16 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=failing
08:16 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups-legacy
08:16 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-hiveserde-tables-in-place-experimental
08:16 INFO [databricks.labs.ucx.install] Creating dashboards...
08:16 DEBUG [databricks.labs.ucx.install] Reading step folder /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/progress...
08:16 DEBUG [databricks.labs.ucx.install] Reading step folder /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/assessment...
08:16 DEBUG [databricks.labs.ucx.install] Reading step folder /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/migration...
08:16 DEBUG [databricks.labs.ucx.install] Reading step folder /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/views...
08:16 INFO [databricks.labs.ucx.install] Creating dashboard in /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/progress/main...
08:16 INFO [databricks.labs.ucx.install] Creating dashboard in /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/assessment/estimates...
08:16 INFO [databricks.labs.ucx.install] Creating dashboard in /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/assessment/main...
08:16 INFO [databricks.labs.ucx.install] Creating dashboard in /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/assessment/CLOUD_ENV...
08:16 INFO [databricks.labs.ucx.install] Creating dashboard in /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/assessment/interactive...
08:16 INFO [databricks.labs.ucx.install] Creating dashboard in /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/migration/main...
08:16 INFO [databricks.labs.ucx.install] Creating dashboard in /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/migration/groups...
08:16 INFO [databricks.labs.ucx.installer.mixins] Fetching warehouse_id from a config
08:16 INFO [databricks.labs.ucx.installer.mixins] Fetching warehouse_id from a config
08:16 INFO [databricks.labs.ucx.installer.mixins] Fetching warehouse_id from a config
08:16 INFO [databricks.labs.ucx.installer.mixins] Fetching warehouse_id from a config
08:16 INFO [databricks.labs.ucx.installer.mixins] Fetching warehouse_id from a config
08:16 INFO [databricks.labs.ucx.installer.mixins] Fetching warehouse_id from a config
08:16 INFO [databricks.labs.ucx.installer.mixins] Fetching warehouse_id from a config
08:16 INFO [databricks.labs.ucx.install] Installation completed successfully! Please refer to the https://DATABRICKS_HOST/#workspace/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.shLr/README for the next steps.
08:16 INFO [databricks.labs.ucx.progress.install] Installation completed successfully!
08:16 DEBUG [databricks.labs.ucx.installer.workflows] starting assessment job: https://DATABRICKS_HOST#job/181773325001811
08:16 INFO [databricks.labs.ucx.installer.workflows] Started assessment job: https://DATABRICKS_HOST#job/181773325001811/runs/963140587285874
08:16 DEBUG [databricks.labs.ucx.installer.workflows] Validating assessment workflow: https://DATABRICKS_HOST#job/181773325001811
08:16 INFO [databricks.labs.ucx.installer.workflows] Identified a run in progress waiting for run completion
08:36 INFO [databricks.labs.ucx.install] Deleting UCX v0.56.1+720250226081557 from https://DATABRICKS_HOST
08:36 INFO [databricks.labs.ucx.install] Deleting inventory database dummy_srqwo
08:36 INFO [databricks.labs.ucx.installer.workflows] Removing job_id=999098573184651, as it is no longer needed
08:36 INFO [databricks.labs.ucx.installer.workflows] Removing job_id=665876467594703, as it is no longer needed
08:36 INFO [databricks.labs.ucx.installer.workflows] Removing job_id=1102832905983516, as it is no longer needed
08:36 INFO [databricks.labs.ucx.installer.workflows] Removing job_id=552911978145919, as it is no longer needed
08:36 INFO [databricks.labs.ucx.installer.workflows] Removing job_id=833706291986884, as it is no longer needed
08:36 INFO [databricks.labs.ucx.installer.workflows] Removing job_id=383946472951583, as it is no longer needed
08:36 INFO [databricks.labs.ucx.installer.workflows] Removing job_id=426283512907378, as it is no longer needed
08:36 INFO [databricks.labs.ucx.installer.workflows] Removing job_id=316423612288756, as it is no longer needed
08:36 INFO [databricks.labs.ucx.installer.workflows] Removing job_id=835652063982350, as it is no longer needed
08:36 INFO [databricks.labs.ucx.installer.workflows] Removing job_id=181773325001811, as it is no longer needed
08:36 INFO [databricks.labs.ucx.installer.workflows] Removing job_id=129827696292019, as it is no longer needed
08:36 INFO [databricks.labs.ucx.installer.workflows] Removing job_id=764530643428928, as it is no longer needed
08:36 INFO [databricks.labs.ucx.installer.workflows] Removing job_id=812274787003688, as it is no longer needed
08:36 INFO [databricks.labs.ucx.install] Deleting cluster policy
08:36 INFO [databricks.labs.ucx.install] Deleting secret scope
08:36 INFO [databricks.labs.ucx.install] UnInstalling UCX complete
[gw4] linux -- Python 3.10.16 /home/runner/work/ucx/ucx/.venv/bin/python
❌ test_table_migration_convert_manged_to_external: AssertionError: Workflow failed: assessment (21m34.234s) 
... (skipped 7096 bytes)
onftest] Waiting for clusters to start...
08:16 INFO [databricks.labs.ucx.install] Installing UCX v0.56.1+720250226081608
08:16 INFO [databricks.labs.ucx.install] Creating ucx schemas...
08:16 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=remove-workspace-local-backup-groups
08:16 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=assessment
08:16 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=scan-tables-in-mounts-experimental
08:16 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups-legacy
08:16 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-hiveserde-tables-in-place-experimental
08:16 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-tables-ctas
08:16 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables-in-mounts-experimental
08:16 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-data-reconciliation
08:16 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migration-progress-experimental
08:16 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=validate-groups-permissions
08:16 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups
08:16 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=failing
08:16 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables
08:16 INFO [databricks.labs.ucx.install] Creating dashboards...
08:16 DEBUG [databricks.labs.ucx.install] Reading step folder /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/progress...
08:16 DEBUG [databricks.labs.ucx.install] Reading step folder /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/assessment...
08:16 DEBUG [databricks.labs.ucx.install] Reading step folder /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/migration...
08:16 DEBUG [databricks.labs.ucx.install] Reading step folder /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/views...
08:16 INFO [databricks.labs.ucx.install] Creating dashboard in /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/progress/main...
08:16 INFO [databricks.labs.ucx.install] Creating dashboard in /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/assessment/estimates...
08:16 INFO [databricks.labs.ucx.install] Creating dashboard in /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/assessment/main...
08:16 INFO [databricks.labs.ucx.install] Creating dashboard in /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/assessment/CLOUD_ENV...
08:16 INFO [databricks.labs.ucx.install] Creating dashboard in /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/assessment/interactive...
08:16 INFO [databricks.labs.ucx.install] Creating dashboard in /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/migration/main...
08:16 INFO [databricks.labs.ucx.install] Creating dashboard in /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/migration/groups...
08:16 INFO [databricks.labs.ucx.installer.mixins] Fetching warehouse_id from a config
08:16 INFO [databricks.labs.ucx.installer.mixins] Fetching warehouse_id from a config
08:16 INFO [databricks.labs.ucx.installer.mixins] Fetching warehouse_id from a config
08:16 INFO [databricks.labs.ucx.installer.mixins] Fetching warehouse_id from a config
08:16 INFO [databricks.labs.ucx.installer.mixins] Fetching warehouse_id from a config
08:16 INFO [databricks.labs.ucx.installer.mixins] Fetching warehouse_id from a config
08:16 INFO [databricks.labs.ucx.installer.mixins] Fetching warehouse_id from a config
08:16 INFO [databricks.labs.ucx.install] Installation completed successfully! Please refer to the https://DATABRICKS_HOST/#workspace/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.dYKq/README for the next steps.
08:16 INFO [databricks.labs.ucx.progress.install] Installation completed successfully!
08:16 DEBUG [databricks.labs.ucx.installer.workflows] starting assessment job: https://DATABRICKS_HOST#job/152481204552255
08:16 INFO [databricks.labs.ucx.installer.workflows] Started assessment job: https://DATABRICKS_HOST#job/152481204552255/runs/930263631438590
08:16 DEBUG [databricks.labs.ucx.installer.workflows] Validating assessment workflow: https://DATABRICKS_HOST#job/152481204552255
08:16 INFO [databricks.labs.ucx.installer.workflows] Identified a run in progress waiting for run completion
08:36 INFO [databricks.labs.ucx.install] Deleting UCX v0.56.1+720250226081608 from https://DATABRICKS_HOST
08:36 INFO [databricks.labs.ucx.install] Deleting inventory database dummy_szjwc
08:37 INFO [databricks.labs.ucx.installer.workflows] Removing job_id=920743581926253, as it is no longer needed
08:37 INFO [databricks.labs.ucx.installer.workflows] Removing job_id=152481204552255, as it is no longer needed
08:37 INFO [databricks.labs.ucx.installer.workflows] Removing job_id=352421448605194, as it is no longer needed
08:37 INFO [databricks.labs.ucx.installer.workflows] Removing job_id=777104403468705, as it is no longer needed
08:37 INFO [databricks.labs.ucx.installer.workflows] Removing job_id=1040659100758768, as it is no longer needed
08:37 INFO [databricks.labs.ucx.installer.workflows] Removing job_id=331590844544533, as it is no longer needed
08:37 INFO [databricks.labs.ucx.installer.workflows] Removing job_id=108741254002465, as it is no longer needed
08:37 INFO [databricks.labs.ucx.installer.workflows] Removing job_id=246723095108275, as it is no longer needed
08:37 INFO [databricks.labs.ucx.installer.workflows] Removing job_id=791091373109078, as it is no longer needed
08:37 INFO [databricks.labs.ucx.installer.workflows] Removing job_id=74114877507466, as it is no longer needed
08:37 INFO [databricks.labs.ucx.installer.workflows] Removing job_id=794722109489940, as it is no longer needed
08:37 INFO [databricks.labs.ucx.installer.workflows] Removing job_id=815515972229026, as it is no longer needed
08:37 INFO [databricks.labs.ucx.installer.workflows] Removing job_id=173642169201982, as it is no longer needed
08:37 INFO [databricks.labs.ucx.install] Deleting cluster policy
08:37 INFO [databricks.labs.ucx.install] Deleting secret scope
08:37 INFO [databricks.labs.ucx.install] UnInstalling UCX complete
[gw5] linux -- Python 3.10.16 /home/runner/work/ucx/ucx/.venv/bin/python

Running from acceptance #8402

@JCZuurmond JCZuurmond force-pushed the feat/pass-table-migration-index-to-workflow-linter branch from 0f173cd to 84a4f7d Compare February 25, 2025 16:01
@JCZuurmond JCZuurmond assigned JCZuurmond and pritishpai and unassigned JCZuurmond Mar 4, 2025
@JCZuurmond JCZuurmond marked this pull request as draft March 4, 2025 07:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@asnare asnare Awaiting requested review from asnare

At least 1 approving review is required to merge this pull request.

Assignees

@pritishpai pritishpai

Labels
migrate/code Abstract Syntax Trees and other dark magic
Projects
UCX
Status: Todo
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[TODO] bring back self.tables_migrator.index()
2 participants
@JCZuurmond @pritishpai