fix: add schema updates with no key properties #52
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This was referenced Jun 11, 2023
pnadolny13
added a commit
that referenced
this pull request
Jun 14, 2023
Closes #48 Sqlalchemy handles some of the quoting for reserved words but we also build sql manually so those dont get handled. Also when doing column diffing like to decided whether a column exists or not we need to use uppercase column names because we created them as uppercase with quotes so case matters. I found a bug with the COPY logic when the stream has schema updates. Logged it in #53 and draft PR in #52. Once that is fixed we can add a schema change to the test records with no key properties so we assert alters work with reserved words when no key properties are being used.
|
@pnadolny13 this is interesting. Sounds to me like a bug with the flushing mechanism - we should be flushing the old stream completely before creating a new stream (including stage etc.) on the arrival of a new Schema 🤔 |
|
Hitting something pretty similar to this but slightly different. I ran with this exactly singer data and wanted to record the current stack trace just to help someone else out if they are getting the same thing |
I wonder if the issue here is #116 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
As part of #51 I wanted to add a test to assert schema changes when reserved words were used and no key properties were present so the COPY logic would get invoked. It failed and I realized that it would have been failing before my changes in that PR.
This PR adds the test case so we can figure out how to fix it. The problem I saw was that the file in the internal stage matched the initial schema record but by the time it was being loaded into the table the second schema message had already altered the table to the new schema. Ultimately leading to an error related to a mismatch in columns in the table vs stage file columns.