source-genesys: new connector #2136
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Currently, only the OAuth2 Authorization Code grant type flow is supported. Upcoming SaaS connectors require more flexibility & can use different OAuth2 grant type flows. For example, the Genesys connector needs a user-configured domain to perform authentication and get an access token. Our current OAuth2 framework only supports access token URLs that don't contain user-configured values, and adding that support would be a larger effort than we want to take on right now. With the Client Credentials grant type flow, we can inject a user-configured value in the access token URL at runtime to support this use case.
Alex-Bair
force-pushed
the
bair/source-genesys
branch
from
a947950
to
60aded2
Compare
Alex-Bair
force-pushed
the
bair/source-genesys
branch
2 times, most recently
from
a94e1e0
to
330204e
Compare
This is a minimal connector for Genesys, containing just the `users` and `conversations` streams. `users` is a simple snapshot since there is no way to request updated users. `conversations` is incremental and leverage two different endpoints. A real-time endpoint is used to get created conversations incrementally. This real-time endpoint is not appropriate to get updates (we can only specify time ranges for created conversations), so we also attempt to use the delayed, asynchronous analytics job endpoint to get conversations that have ended in the last day. This is not meant to capture all ended conversations as there are scenarios where we would miss capturing these updates (ex: no conversations are created for a while but conversations are ended), and a backfill will be required to fully capture all ended conversations.
The code for retrieving conversations in real-time from the query endpoint is left in the previous commit in case we want to add this functionality later when the CDK has the capability to have a delayed stream / a second cursor.
Alex-Bair
force-pushed
the
bair/source-genesys
branch
from
330204e
to
58deca8
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description:
This PR introduces a new connector
source-genesyswith two streams.The main changes in this PR are:
estuary-cdkto support the OAuth2 client credentials grant type flow as detailed per the OAuth 2.0 standard.source-genesysconnector with the following two streams:users- snapshotconversations- incrementala. The asynchronous analytics job endpoint is used for incremental updates. It is not updated in real-time, and data availability can be delayed by hours to a full day (ex: the delay's been ~13 hours during my testing). However, this is the only reliable way to get updated conversations, so it's used.
b. The query endpoint is updated in real-time, but is only appropriate for retrieving conversations when they're created - we can't ask this endpoint for updated conversations. An implementation using this query endpoint is left in a previous commit in case we want to use it later if/when the CDK can support delayed streams or multiple cursors per stream.
Workflow steps:
(How does one use this feature, and how has it changed)
Documentation links affected:
Documentation will need created for
source-genesys. I'll link the doc creation PR once it is ready.Notes for reviewers:
Tested on a local stack. Confirmed:
userssnapshots work as intended and retrieve all expected data.conversationsbackfills and incremental replication work as intended and retrieve all expected data.I used a trial Genesys account for initial testing & setup. Since these trial accounts expire after a month, I did not include a capture snapshot.
This change is