DBZ-6721 & DBZ-6722 Add transforms: local vgtid, remove field, and filter transaction topic #194
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
twthorn
changed the title
twthorn
changed the title
|
Hi @twthorn, thanks for your contribution. Please prefix the commit message(s) with the DBZ-xxx JIRA issue key. |
|
@jpechane all updated with refactor to move from core to here, tests are all passing, ready for review |
twthorn
changed the title
|
@twthorn LGTM, thanks. Could you please rebase the PR on the latest main and remove merge commit(s)? When cleaned up the PR is ready to be merged. |
|
@jpechane thanks for the review, updated! |
|
@twthorn Applied, thanks |
|
Could you please provide a short update of vitess connector docs describing the new capability? Thanks |
|
PR to update docs debezium/debezium#5634 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
DBZ-6721
DBZ-6722
Add three new transforms
Filter Transaction Topic
For connectors where the volume is high (transaction topic is massive), we can drop all messages matching the schema
Remove Field
Generic config for specifying field names (separated by dot '.' for nesting) to remove fields. Can be used to drop any field, in this case we use for transaction.id which can be very large in vitess (contains all shards gtid).
Local VGTID
Update VGTID field to only contain vgtid for the shard that the change happened in.
We add this local vgtid transformation (add a hashmap so we don't have to iterate over all shards to do this).
We need to keep the vgtid global for proper offset/state storage so therefore it must be done only at the end as SMT before publishing the message (and does not affect source offset of the message).
Itests
Also add itests for all these functionalities together for how they can be used to receive transaction order metadata without massively increasing data volume produced.