Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

sqlcapture: include collection schema with a slash in collection name #1907

Merged
merged 2 commits into from
Jan 7, 2025
Merged

sqlcapture: include collection schema with a slash in collection name #1907

merged 2 commits into from
Jan 7, 2025

Conversation

mdibaiee
Copy link
Member

@mdibaiee mdibaiee commented Sep 10, 2024 •
edited
Loading

Description:

The actual code change is the first commit, the second commit is updating all the test snapshots

Part of estuary/flow#1583

Workflow steps:

(How does one use this feature, and how has it changed)

Documentation links affected:

(list any documentation links that you created, or existing ones that you've identified as needing updates, along with a brief description)

Notes for reviewers:

(anything that might help someone review this PR)

This change is Reviewable

@mdibaiee mdibaiee force-pushed the mahdi/capture-schemas branch from f2265e8 to d82a2f1 Compare September 10, 2024 14:59
@mdibaiee mdibaiee changed the title Mahdi/capture schemas sqlcapture: update test snapshots for new collection names Sep 10, 2024
@mdibaiee mdibaiee added the change:planned This is a planned change label Sep 10, 2024
@mdibaiee mdibaiee changed the title sqlcapture: update test snapshots for new collection names sqlcapture: include collection schema with a slash in collection name Sep 10, 2024
@mdibaiee mdibaiee force-pushed the mahdi/capture-schemas branch from d82a2f1 to 6aa784d Compare September 11, 2024 09:58
@mdibaiee mdibaiee mentioned this pull request Sep 30, 2024
Open
@mdibaiee mdibaiee force-pushed the mahdi/capture-schemas branch 2 times, most recently from b491a56 to 4b7667a Compare January 2, 2025 17:05
@mdibaiee mdibaiee marked this pull request as ready for review January 2, 2025 17:05
@mdibaiee mdibaiee force-pushed the mahdi/capture-schemas branch from 4b7667a to 458ed55 Compare January 2, 2025 17:23
williamhbaker
williamhbaker approved these changes Jan 3, 2025
View reviewed changes
Copy link
Member

@williamhbaker williamhbaker left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Not sure if the remaining source-sqlserver test failures are related to these changes or not

@mdibaiee
Copy link
Member Author

mdibaiee commented Jan 6, 2025

@williamhbaker it's strange because the test passes on my local on this branch

connectors git:(mahdi/capture-schemas) TEST_DATABASE=yes UPDATE_SNAPSHOTS=true go test -v ./source-sqlserver -timeout 99999h -run TestAlterationAddColumn
=== RUN   TestAlterationAddColumn
time="2025-01-06T20:39:46Z" level=info msg="opening control connection" addr="127.0.0.1:1433" user=sa
=== RUN   TestAlterationAddColumn/Off
time="2025-01-06T20:39:47Z" level=info msg="initializing connector" address="127.0.0.1:1433" user=flow_capture
time="2025-01-06T20:39:47Z" level=info msg="initializing connector" address="127.0.0.1:1433" user=flow_capture
time="2025-01-06T20:39:47Z" level=info msg="queried database version" major=16 minor=0 version=16.0.4165.4
time="2025-01-06T20:39:47Z" level=info msg="discovering tables"
time="2025-01-06T20:39:47Z" level=info msg="binding added to capture" stateKey="dbo%2Ftest_AlterationAddColumn_Off_39517707"
time="2025-01-06T20:39:47Z" level=info msg="all bindings are new, resetting replication cursor"
time="2025-01-06T20:39:47Z" level=info msg="streaming to fence" fenceAfter=0s
time="2025-01-06T20:39:49Z" level=info msg="processed replication events" events=4
time="2025-01-06T20:39:49Z" level=info msg="activating replication for stream" mode= stream=dbo.test_alterationaddcolumn_off_39517707
time="2025-01-06T20:39:49Z" level=info msg="autoselected precise backfill mode" stream=dbo.test_alterationaddcolumn_off_39517707
time="2025-01-06T20:39:49Z" level=info msg="backfilling streams" count=1 selected=dbo.test_alterationaddcolumn_off_39517707
time="2025-01-06T20:39:49Z" level=info msg="processed backfill rows" rows=2 stream=dbo.test_alterationaddcolumn_off_39517707
time="2025-01-06T20:39:49Z" level=info msg="backfill completed" stream=dbo.test_alterationaddcolumn_off_39517707
time="2025-01-06T20:39:49Z" level=info msg="streaming to fence" fenceAfter=0s
time="2025-01-06T20:39:54Z" level=info msg="processed replication events" events=2
time="2025-01-06T20:39:55Z" level=info msg="initializing connector" address="127.0.0.1:1433" user=flow_capture
time="2025-01-06T20:39:55Z" level=info msg="queried database version" major=16 minor=0 version=16.0.4165.4
time="2025-01-06T20:39:55Z" level=info msg="discovering tables"
time="2025-01-06T20:39:55Z" level=info msg="streaming to fence" fenceAfter=0s
time="2025-01-06T20:39:59Z" level=info msg="processed replication events" events=6
time="2025-01-06T20:39:59Z" level=info msg="initializing connector" address="127.0.0.1:1433" user=flow_capture
time="2025-01-06T20:39:59Z" level=info msg="queried database version" major=16 minor=0 version=16.0.4165.4
time="2025-01-06T20:39:59Z" level=info msg="discovering tables"
time="2025-01-06T20:39:59Z" level=info msg="streaming to fence" fenceAfter=0s
time="2025-01-06T20:40:04Z" level=info msg="processed replication events" events=6
time="2025-01-06T20:40:05Z" level=info msg="initializing connector" address="127.0.0.1:1433" user=flow_capture
time="2025-01-06T20:40:05Z" level=info msg="queried database version" major=16 minor=0 version=16.0.4165.4
time="2025-01-06T20:40:05Z" level=info msg="discovering tables"
time="2025-01-06T20:40:05Z" level=info msg="streaming to fence" fenceAfter=0s
time="2025-01-06T20:40:09Z" level=info msg="processed replication events" events=6
time="2025-01-06T20:40:09Z" level=info msg="initializing connector" address="127.0.0.1:1433" user=flow_capture
time="2025-01-06T20:40:09Z" level=info msg="queried database version" major=16 minor=0 version=16.0.4165.4
time="2025-01-06T20:40:09Z" level=info msg="discovering tables"
time="2025-01-06T20:40:09Z" level=info msg="streaming to fence" fenceAfter=0s
time="2025-01-06T20:40:14Z" level=info msg="processed replication events" events=6
=== RUN   TestAlterationAddColumn/Manual
time="2025-01-06T20:40:15Z" level=info msg="initializing connector" address="127.0.0.1:1433" user=flow_capture
time="2025-01-06T20:40:15Z" level=info msg="initializing connector" address="127.0.0.1:1433" user=flow_capture
time="2025-01-06T20:40:15Z" level=info msg="queried database version" major=16 minor=0 version=16.0.4165.4
time="2025-01-06T20:40:15Z" level=info msg="discovering tables"
time="2025-01-06T20:40:15Z" level=info msg="binding added to capture" stateKey="dbo%2Ftest_AlterationAddColumn_Manual_22713060"
time="2025-01-06T20:40:15Z" level=info msg="all bindings are new, resetting replication cursor"
time="2025-01-06T20:40:15Z" level=info msg="streaming to fence" fenceAfter=0s
time="2025-01-06T20:40:19Z" level=info msg="processed replication events" events=4
time="2025-01-06T20:40:19Z" level=info msg="activating replication for stream" mode= stream=dbo.test_alterationaddcolumn_manual_22713060
time="2025-01-06T20:40:19Z" level=info msg="autoselected precise backfill mode" stream=dbo.test_alterationaddcolumn_manual_22713060
time="2025-01-06T20:40:19Z" level=info msg="backfilling streams" count=1 selected=dbo.test_alterationaddcolumn_manual_22713060
time="2025-01-06T20:40:19Z" level=info msg="processed backfill rows" rows=2 stream=dbo.test_alterationaddcolumn_manual_22713060
time="2025-01-06T20:40:19Z" level=info msg="backfill completed" stream=dbo.test_alterationaddcolumn_manual_22713060
time="2025-01-06T20:40:19Z" level=info msg="streaming to fence" fenceAfter=0s
time="2025-01-06T20:40:24Z" level=info msg="processed replication events" events=2
time="2025-01-06T20:40:27Z" level=info msg="initializing connector" address="127.0.0.1:1433" user=flow_capture
time="2025-01-06T20:40:27Z" level=info msg="queried database version" major=16 minor=0 version=16.0.4165.4
time="2025-01-06T20:40:27Z" level=info msg="discovering tables"
time="2025-01-06T20:40:27Z" level=info msg="streaming to fence" fenceAfter=0s
time="2025-01-06T20:40:29Z" level=info msg="processed replication events" events=6
time="2025-01-06T20:40:29Z" level=info msg="initializing connector" address="127.0.0.1:1433" user=flow_capture
time="2025-01-06T20:40:29Z" level=info msg="queried database version" major=16 minor=0 version=16.0.4165.4
time="2025-01-06T20:40:29Z" level=info msg="discovering tables"
time="2025-01-06T20:40:29Z" level=info msg="streaming to fence" fenceAfter=0s
time="2025-01-06T20:40:34Z" level=info msg="processed replication events" events=6
time="2025-01-06T20:40:39Z" level=info msg="initializing connector" address="127.0.0.1:1433" user=flow_capture
time="2025-01-06T20:40:39Z" level=info msg="queried database version" major=16 minor=0 version=16.0.4165.4
time="2025-01-06T20:40:39Z" level=info msg="discovering tables"
time="2025-01-06T20:40:39Z" level=info msg="streaming to fence" fenceAfter=0s
time="2025-01-06T20:40:39Z" level=info msg="processed replication events" events=6
time="2025-01-06T20:40:39Z" level=info msg="initializing connector" address="127.0.0.1:1433" user=flow_capture
time="2025-01-06T20:40:40Z" level=info msg="queried database version" major=16 minor=0 version=16.0.4165.4
time="2025-01-06T20:40:40Z" level=info msg="discovering tables"
time="2025-01-06T20:40:40Z" level=info msg="streaming to fence" fenceAfter=0s
time="2025-01-06T20:40:45Z" level=info msg="processed replication events" events=6
=== RUN   TestAlterationAddColumn/Automatic
time="2025-01-06T20:40:48Z" level=info msg="initializing connector" address="127.0.0.1:1433" user=flow_capture
time="2025-01-06T20:40:48Z" level=info msg="initializing connector" address="127.0.0.1:1433" user=flow_capture
time="2025-01-06T20:40:48Z" level=info msg="queried database version" major=16 minor=0 version=16.0.4165.4
time="2025-01-06T20:40:48Z" level=info msg="discovering tables"
time="2025-01-06T20:40:48Z" level=info msg="binding added to capture" stateKey="dbo%2Ftest_AlterationAddColumn_Automatic_81310450"
time="2025-01-06T20:40:48Z" level=info msg="all bindings are new, resetting replication cursor"
time="2025-01-06T20:40:48Z" level=info msg="streaming to fence" fenceAfter=0s
time="2025-01-06T20:40:50Z" level=info msg="processed replication events" events=4
time="2025-01-06T20:40:50Z" level=info msg="activating replication for stream" mode= stream=dbo.test_alterationaddcolumn_automatic_81310450
time="2025-01-06T20:40:50Z" level=info msg="autoselected precise backfill mode" stream=dbo.test_alterationaddcolumn_automatic_81310450
time="2025-01-06T20:40:50Z" level=info msg="backfilling streams" count=1 selected=dbo.test_alterationaddcolumn_automatic_81310450
time="2025-01-06T20:40:50Z" level=info msg="processed backfill rows" rows=2 stream=dbo.test_alterationaddcolumn_automatic_81310450
time="2025-01-06T20:40:50Z" level=info msg="backfill completed" stream=dbo.test_alterationaddcolumn_automatic_81310450
time="2025-01-06T20:40:50Z" level=info msg="streaming to fence" fenceAfter=0s
time="2025-01-06T20:40:55Z" level=info msg="processed replication events" events=2
time="2025-01-06T20:40:56Z" level=info msg="initializing connector" address="127.0.0.1:1433" user=flow_capture
time="2025-01-06T20:40:56Z" level=info msg="queried database version" major=16 minor=0 version=16.0.4165.4
time="2025-01-06T20:40:56Z" level=info msg="discovering tables"
time="2025-01-06T20:40:56Z" level=info msg="streaming to fence" fenceAfter=0s
time="2025-01-06T20:41:00Z" level=info msg="processed replication events" events=6
time="2025-01-06T20:41:00Z" level=info msg="initializing connector" address="127.0.0.1:1433" user=flow_capture
time="2025-01-06T20:41:00Z" level=info msg="queried database version" major=16 minor=0 version=16.0.4165.4
time="2025-01-06T20:41:00Z" level=info msg="discovering tables"
time="2025-01-06T20:41:00Z" level=info msg="streaming to fence" fenceAfter=0s
time="2025-01-06T20:41:00Z" level=info msg="created new capture instance for table" instance=dbo_test_AlterationAddColumn_Automatic_81310450_436D853A_1736196060 schema=dbo table=test_AlterationAddColumn_Automatic_81310450
time="2025-01-06T20:41:05Z" level=info msg="processed replication events" events=6
time="2025-01-06T20:41:06Z" level=info msg="initializing connector" address="127.0.0.1:1433" user=flow_capture
time="2025-01-06T20:41:06Z" level=info msg="queried database version" major=16 minor=0 version=16.0.4165.4
time="2025-01-06T20:41:06Z" level=info msg="discovering tables"
time="2025-01-06T20:41:06Z" level=info msg="streaming to fence" fenceAfter=0s
time="2025-01-06T20:41:06Z" level=info msg="deleted obsolete capture instance" instance=dbo_test_AlterationAddColumn_Automatic_81310450 schema=dbo table=test_AlterationAddColumn_Automatic_81310450
time="2025-01-06T20:41:10Z" level=info msg="processed replication events" events=6
time="2025-01-06T20:41:10Z" level=info msg="initializing connector" address="127.0.0.1:1433" user=flow_capture
time="2025-01-06T20:41:10Z" level=info msg="queried database version" major=16 minor=0 version=16.0.4165.4
time="2025-01-06T20:41:10Z" level=info msg="discovering tables"
time="2025-01-06T20:41:10Z" level=info msg="streaming to fence" fenceAfter=0s
time="2025-01-06T20:41:10Z" level=info msg="created new capture instance for table" instance=dbo_test_AlterationAddColumn_Automatic_81310450_436D853A_1736196070 schema=dbo table=test_AlterationAddColumn_Automatic_81310450
time="2025-01-06T20:41:15Z" level=info msg="processed replication events" events=6
--- PASS: TestAlterationAddColumn (89.05s)
    --- PASS: TestAlterationAddColumn/Off (27.98s)
    --- PASS: TestAlterationAddColumn/Manual (32.83s)
    --- PASS: TestAlterationAddColumn/Automatic (28.22s)
PASS
ok      github.com/estuary/connectors/source-sqlserver  89.372s

@mdibaiee mdibaiee force-pushed the mahdi/capture-schemas branch from 458ed55 to 8f5d255 Compare January 6, 2025 21:22
@williamhbaker
Copy link
Member

Might just need a rebase

@mdibaiee
Copy link
Member Author

mdibaiee commented Jan 6, 2025

@williamhbaker still succeeds locally after a rebase (cleared test cache with -count 1) and fails on CI...

@mdibaiee
Copy link
Member Author

mdibaiee commented Jan 6, 2025

Going to run the whole test suite, wondering of if there are side-effects of other tests affecting this one tesst

@mdibaiee mdibaiee force-pushed the mahdi/capture-schemas branch from 8f5d255 to c9ff348 Compare January 6, 2025 23:56
@mdibaiee
Copy link
Member Author

mdibaiee commented Jan 7, 2025

I haven't been able to get the tests to fail locally, I'll merge, let's see what happens on main branch

@mdibaiee mdibaiee merged commit a7fa34c into main Jan 7, 2025
51 of 54 checks passed
@mdibaiee mdibaiee deleted the mahdi/capture-schemas branch January 7, 2025 16:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@williamhbaker williamhbaker williamhbaker approved these changes

@willdonnelly willdonnelly Awaiting requested review from willdonnelly

@Alex-Bair Alex-Bair Awaiting requested review from Alex-Bair

Assignees
No one assigned
Labels
change:planned This is a planned change
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@mdibaiee @williamhbaker