source-twilio: capture messages updates based on lookback_window
#2225
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…treams No user's tasks are currently using the lookback window setting, so I'm removing code that shifts incremental streams' states back based on the lookback window. This is pre-positioning for future work that makes the lookback window affect the date range over which we query Twilio's API instead of mutating streams' states.
Alex-Bair
force-pushed
the
bair/source-twilio-messages-updates
branch
2 times, most recently
from
ba1925a to
10d1ff0
Compare
Alex-Bair
added
the
change:unplanned
Twilio's API only allows us to query messages based on their creation date, not their updated date. Previously, the connector was only getting creates, with the option to use the lookback window to push back the cursor value to re-emit all messages created in the past X minutes. The `messages` stream now uses the lookback window to query Twilio for messages created X minutes in the past, but it only emits messages that have been created or updated after the current cursor value.
Alex-Bair
force-pushed
the
bair/source-twilio-messages-updates
branch
from
10d1ff0 to
91b89ab
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description:
This PR's scope includes:
lookback_windowminutes.lookback_windowfrom affecting themessage_mediastream. Media can't be added to messages after they're created, so it doesn't make sense to check previously created messages for new media.messagesdocuments if they happen within the pastlookback_windowminutes.Additional details:
Twilio's API only allows us to query messages based on their creation date
date_sent, not their updated datedate_updated. Previously, the connector was only getting creates, with the option to use thelookback_windowconfig setting to push back the cursor value, causing all messages created in the pastlookback_windowminutes to get re-captured. Themessagesstream now uses the lookback window to query Twilio for messages createdlookback_windowminutes in the past, but it only captures messages that have been created or updated after the current cursor value.Future work includes using
lookback_windowlike this for the other incremental streams (calls, etc.) as appropriate to capture updates as well as creates. I didn't do those streams in this PR since they aren't the current issue & it'd take some additional time to make sure they work as intended.Workflow steps:
(How does one use this feature, and how has it changed)
Documentation links affected:
(list any documentation links that you created, or existing ones that you've identified as needing updates, along with a brief description)
Notes for reviewers:
Tested on a local stack. Confirmed:
messagescaptures updates to records made in the pastlookback_windowminutes.messagesstill captures creates.message_mediastill functions incrementally.Any users who want to use this functionality should set their
lookback_windowto a value greater than 0. For example, if a user only cares about updates to messages within the past day, they should set their task'slookback_windowto1440.This change is