source-stripe-native: handle responses for children of deleted resources #1982
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Alex-Bair
added
the
change:unplanned
psFried
approved these changes
Sep 26, 2024
| # the requests for the associated child resources fail. Stripe returns a 404 | ||
| # error & a message containing "resource_missing" and "No such" when this happens. | ||
| if err.code == 404 and bool(re.search(MISSING_RESOURCE_REGEX, err.message, re.DOTALL)): | ||
| log.debug(f"Missing resource error for URL {child_url}. Skipping to the next resource.", err) |
There was a problem hiding this comment.
IDK how common it is to get 404 responses, but it might be helpful to log at info or warn levels as long as it wouldn't be too spammy. Feel free to ignore this suggestion if it'd result in a bunch of log spam though.
There was a problem hiding this comment.
Thanks for the suggestion, Phil! I don't anticipate we'd encounter this error all that often. I'll switch it over to warn.
Child streams, like `InvoiceLineItems` handle incremental replication by watching for events on their associated parent resource, then requesting all child resources for that parent. However, since there can be some time between when we process an event & what actually happens in Stripe, it's possible for us to request child resources of a deleted parent. Stripe returns a `404` code and somewhat unique text to indicate this. We can safely skip requesting these child resources when this happens. I noticed this issue happening with `InvoiceLineItems`, so I also removed the "invoice.deleted" event from that stream's `EVENT_TYPES` since we'll just get an error back if we ask for a deleted parent resource's children.
Alex-Bair
force-pushed
the
bair/source-stripe-native-invoice-lines-events
branch
from
d531ba7
to
9ef1a88
Compare
Alex-Bair
deleted the
bair/source-stripe-native-invoice-lines-events
branch
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description:
Child streams, like
InvoiceLineItemsperform incremental replication by watching for events on their associated parent resource, then requesting all child resources for that parent when an event happens. However, since there can be some time between when we process an event & what actually happens in Stripe, it's possible for us to request child resources of a deleted parent. Stripe returns a404code and somewhat unique text to indicate this. We can safely skip requesting these resources when this happens.This situation can happen if a creation/update is quickly followed by a deletion. If the connector is stuck for a while and a backlog of events we haven't processed builds up, there's even more opportunity for us to request deleted resources.
I noticed this issue happening with
InvoiceLineItems, so I also removed the "invoice.deleted" event from that stream'sEVENT_TYPESsince we'll just get an error back if we request a deleted invoice's line items.Workflow steps:
(How does one use this feature, and how has it changed)
Documentation links affected:
(list any documentation links that you created, or existing ones that you've identified as needing updates, along with a brief description)
Notes for reviewers:
Tested on a local stack. Confirmed that requesting children of a deleted parent resource no longer crashes the connector & that the connector continues processing after handling the error.
This change is