Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

estuary-cdk idle backoff tweak & readme #1330

Merged
merged 4 commits into from
Mar 8, 2024
Merged

estuary-cdk idle backoff tweak & readme #1330

merged 4 commits into from
Mar 8, 2024

Commits on Mar 6, 2024

  1. estuary-cdk: incremental tasks are idle if their cursor is very recent

    Some resources always have _some_ level of activity every time they're
    checked, which would ordinarily prevent the resource from ever being
    considered "caught up", preventing the connector from exiting.
    
    If the LogCursor is a date-time, then additionally consider the task to
    be idle if the LogCursor is less-than `interval` old (where interval is
    configured on the ResourceConfig), and sleep until the cursor is
    `interval` old before issuing a successive fetch.
    jgraettinger committed Mar 6, 2024
    Configuration menu
    Copy the full SHA
    ae3ece9 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    807338d View commit details
    Browse the repository at this point in the history

Commits on Mar 7, 2024

  1. Configuration menu
    Copy the full SHA
    1371cc7 View commit details
    Browse the repository at this point in the history
  2. estuary-cdk: FetchPageFn is also an AsyncGenerator

    Much like FetchChangesFn, FetchPageFn is now an AsyncGenerator which
    yields documents, checkpoint-able PageCursors, or completes an
    iteration of a resource.
    
    This refactoring is consistent with recent updates to FetchChangesFn,
    and is motivated by use cases where the fetched "page" could be quite
    large and itself composed of multiple concurrent data fetches,
    implemented as constituent AsyncGenerator instances.
    
    By using an AsyncGenerator, such an implementation can immediately
    yield from across those concurrent fetches as data arrives such that
    data is spilled to disk and memory pressure kept low. Then, when the
    scatter / gather data fetch completes, the entire "page" is
    checkpointed.
    jgraettinger committed Mar 7, 2024
    Configuration menu
    Copy the full SHA
    bdf791f View commit details
    Browse the repository at this point in the history