Phil/rm pattern descriptor #242
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The Standard replication does not work for tables with integer primary keys, because the connector maps integer columns to `type: number` in the json schemas, and changing the type in the schema causes an exception in the connector.
psFried
force-pushed
the
phil/rm-pattern-descriptor
branch
from
5d07e49
to
8b57fae
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
We saw a number of problems with
source-postgres-heroku(Airbyte'ssource-postgres).First was an issue with the endpoint spec schema, which caused our encryption endpoint to fail when creating captures via the UI. This was solved by removing unknown keywords from the spec schema.
Second is that the connector failed to ingest data from a table that had an integer primary key. The Discover RPC returns a schema with
type: numberfor that field, which Flow doesn't allow for primary keys. Changing the type tointegerin the collection schema allows the build to work, but causes an exception in the connector. The second commit here changes the replication method to always use "Xmin", which we thought might allow us to usetype: integer, but it seems to still cause an exception when you usetype: integerin the collection schema. This is tracked upstream in airbytehq#28529So it seems like this connector may just not work until the error with integer fields is fixed upstream. In the meantime, we can either merge this (it's already broken 🤷) or keep it around as a draft / reminder to follow up.