source-intercom-native: new connector

[Alex-Bair]

Description:

This is a new native connector for Intercom.

A few quirks I ran into when developing the connector:

• Intercom provides two separate endpoints for fetching companies. /companies/scroll returns all companies, but only a single "scroll" can be ongoing at a time. /companies/list returns at max 10,000 companies without a limit to how many "lists" can be ongoing. Due to the restrictions with /companies/scroll that would require coordination around which stream is using a "scroll" (and the potential for some other application using a "scroll" that's not the connector), I chose to use /companies/list. We've also yet to see a use case with >100 companies. We can re-evaluate this decision if we come across users that have 10,000+ companies in their Intercom account.
• Although contacts and conversations have a similar /search endpoint, the filtering granularity between them is different.
  • The /conversations/search endpoint filters results by datetime (i.e. down to the second). The connector requests and processes results in ascending order since the first result will always be an updated record we haven't seen yet.
  • The /contacts/search endpoint filters results by date (i.e. down to the day). This means that if sorted in ascending order, the first page(s) of results will almost always contain records we've already seen. To minimize the number of API requests and immediately get any updated results, the contacts stream requests results in descending order.
• An Intercom conversation can have multiple parts within it, which have their own updated_at fields. The connector processes conversation_parts in date_windows since the conversation.updated_at field is the latest conversation_part.updated_at. We don't want to miss parts that were updated after the date window's start and aren't the most recent conversation part. For example, this is a valid conversation object:

conversation = {
  "updated_at": 1698941604,
  "conversation_parts": {
    "conversation_parts": [
      {"updated_at": 1698768292 },
      {"updated_at": 1698941604 }
    ]  
  }
}

• Intercom's flavor of OAuth returns a long-lived access token, so I extended the Estuary CDK to support this.

Note: The updated_at fields were left as int timestamps instead of converting them to AwareDatetimes. There are multiple other int timestamp fields in Intercom resources, so I left updated_at alone so it would align with all the other timestamp fields Intercom populates.

Workflow steps:

(How does one use this feature, and how has it changed)

Documentation links affected:

Documentation should be created/updated for the new connector.

Notes for reviewers:

Tested on a local stack. Confirmed:

• OAuth works with an Intercom OAuth app created in a separate development Intercom account. I'm assuming OAuth will still work when deployed and using our actual prod Intercom OAuth app instead.
• Streams capture the correct amount of data as far as I can tell. (conversation_parts is difficult to verify without manually sending tons of API requests).
• Collections can be materialized into a dockerized PostgreSQL without error.
• The conversation_parts stream does not check every conversation in the user's Intercom account for updated parts (an issue we're seeing with the ATF connector). It only checks updated conversations for updated conversation parts, speeding the stream up significantly.

Any users that want to use this new native Intercom connector instead of the existing ATF connector will need to create a new task & collections.

---

This change is 

[williamhbaker]

LGTM