Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

various unrelated small fixes and tweaks #1752

Merged
merged 6 commits into from
Nov 5, 2024
Merged

Commits on Nov 3, 2024

  1. crates/data-plane-controller: adjust defaults and log about DNS propa…

    …gation
    
    Use some concurrency by default, and increase the heartbeat interval.
    
    Log when we're waiting for DNS propagation.
    jgraettinger committed Nov 3, 2024
    Configuration menu
    Copy the full SHA
    5d8444b View commit details
    Browse the repository at this point in the history
  2. flowctl raw bearer-logs: add --since parameter with 1 hour default

    If there are many historical logs in the stream, you can get a lot of
    output when reading from the beginning of time and the operator is
    usually only interested in what's happening right now.
    jgraettinger committed Nov 3, 2024
    Configuration menu
    Copy the full SHA
    678a3ab View commit details
    Browse the repository at this point in the history
  3. go/runtime: ensure we destroy a SQLite store on Open error

    We were not properly tearning down these instances, which lead to
    lingering open RocksDB instances and open file descriptors to removed
    files, preventing disk from being reclaimed.
    jgraettinger committed Nov 3, 2024
    Configuration menu
    Copy the full SHA
    c64e5c5 View commit details
    Browse the repository at this point in the history
  4. gazette: make Router synchronous and lazy, and clear Read header

    Simplify Router by starting a Channel immediately, and instead emphasize
    RPC timeouts rather than connection timeouts.
    
    There are a variety of conditions where connection timeouts don't really
    work, such as when a server has bound its port but is not actively
    serving it. Currently this causes tonic to block indefinitely.
    
    Also, Channel embeds a reconnection behavior if the transport is broken,
    so it's already the case that Channels are connecting in the background
    and we should expect to see corresponding delays.
    
    So, have a single behavior for Channels: they start immediately but may
    have connection delay in the background.
    
    - Back out connection-readiness route selection in favor of random balancing.
    - Distinguish local vs non-local Channels: in the future, we should use this
      to implement selective compression over non-local Channels.
    
    Also, upon an error a Read stream must clear its ReadRequest Header.
    The rationale is that the current Header could point to a Gazette broker
    which has permanently stopped, and connections will time out repeatedly.
    Clearing the Header causes route discovery to restart using
    the base service address.
    jgraettinger committed Nov 3, 2024
    Configuration menu
    Copy the full SHA
    747ecc3 View commit details
    Browse the repository at this point in the history

Commits on Nov 4, 2024

  1. derive-sqlite: improve transform stub generation

    Lead with an identity-transform example SELECT JSON($flow_document)
    
    Continue to emit an example selecting individual fields, but
    semantically disable it and also remove root document projections,
    because these are never desired.
    jgraettinger committed Nov 4, 2024
    Configuration menu
    Copy the full SHA
    876bc9c View commit details
    Browse the repository at this point in the history

Commits on Nov 5, 2024

  1. gazette/runtime: use HTTP/2 keep-alive intervals

    HTTP/2 keep-alive sends a PING frame every interval, and fails the
    connection of the peer doesn't respond in time. This verifies the
    end-to-end health of the HTTP/2 transport and catches issues like
    servers which have bound sockets but aren't actively listening.
    
    Also using HTTP/2 keep-alive when connecting to local containers. We've
    observed that `podman` can fail in ways that leave the reactor believing
    it has an established connection to flow-connector-init, even though the
    container has failed and the network namespace has been torn down.
    jgraettinger committed Nov 5, 2024
    Configuration menu
    Copy the full SHA
    c703fa6 View commit details
    Browse the repository at this point in the history