Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

first steps towards data-plane-gateway deprecation #1628

Merged
merged 8 commits into from
Sep 20, 2024
Merged

first steps towards data-plane-gateway deprecation #1628

merged 8 commits into from
Sep 20, 2024

Commits on Sep 20, 2024

  1. agent: add additional data_planes columns 

    Track managed data-plane metadata that users care about.

Also add an `enable_l2` toggle to toggle whether data-planes are
included in L2 roll-ups.
    @jgraettinger
    jgraettinger committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    8ebbe32 View commit details
    Browse the repository at this point in the history

  2. agent: refator out Snapshot into a separate module 

    Add `evaluate()` for evaluation of a generic policy, while encapsulating
Snapshot refresh and retry semantics. We'll use this shortly for
additional authorization APIs.
    @jgraettinger
    jgraettinger committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    ddb002b View commit details
    Browse the repository at this point in the history

  3. tables: add UserGrants and refine transitive role search 

    Add UserGrants table.

Refactor RBAC search into a joint search that's generalized over
both user and role grants.
    @jgraettinger
    jgraettinger committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    2974df6 View commit details
    Browse the repository at this point in the history

  4. agent: add /authorize/user/task and /authorize/user/collection ro… 

    …utes

`/authorize/user/task` enables UI shard listings/status and retrieval
of task logs, as well as access to private connector networking.

`/authorize/user/collection` enables UI journal listing and data preview.

Both offer temporary support for the current data-plane-gateway,
which implements legacy authorization checks using claimed prefixes.

Also introduce an address rewrite mechanism for mapping an internal
data-plane legacy service address into the data-plane-gateway address in
external call contexts.

Issue #1627
    @jgraettinger
    jgraettinger committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    244345a View commit details
    Browse the repository at this point in the history

  5. gazette: refactor Router to make clients cheap to clone 

    Don't configure Router with a default service address.

Instead, journal and shard Client instances are configured with a
default service address and metadata which is _used_ by Router when
picking a route.

This makes it possible to cheaply clone Client instances and give each a
different service address and authorization header, while still using
the same underlying pool of gRPC connections.
    @jgraettinger
    jgraettinger committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    030133f View commit details
    Browse the repository at this point in the history

  6. flowctl: refactor config and support new collection & task authorizat… 

    …ions

This change introduces the agent API to `flowctl`, which is the
proverbial straw which motivated a deeper refactor of flowctl
configuration.

As a headline feature, `flowctl` supports the new task and collection
authorization APIs and uses them in support of serving existing
subcommands for reading collections, previews, and read ops logs or
stats.

Clean up management of access and refresh tokens by obtaining access
tokens or generating refresh tokens prior to calling into a particular
sub-command. Preserve the ability to run `flowctl` in an unauthenticated
mode.

Make it easier to use `flowctl` against a local stack by introducing
alternative defaults when running under a "local" profile.

Also fix handling of single-use refresh tokens, where we must retain the
updated secret after using it to generate a new access token. We could
now consider having `flowctl` create single-use refresh tokens rather
than multi-use ones, but I didn't want to take that step just yet.

Also fix mis-ordering of output when reading journals.

Also fix OffsetNotYetAvailable error when reading a journal in non-blocking mode.

Issue #1627
    @jgraettinger
    jgraettinger committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    e912244 View commit details
    Browse the repository at this point in the history

  7. go/network: refactored and updated connector networking feature 

    Move connector networking entirely into this repo, from the legacy
data-plane-gatweay repo, and significantly retool it along the way to:

* Improve latency and throughput of HTTP reverse-proxy cases,
  by allowing the reverse proxy to use multiple pooled connections
  built atop network proxy RPCs with reasonable idle timeouts.

  This improves concurrency as many HTTP/2 requests can be in flight at
  once, and improves latency to the end user by ammortizing connections
  to reduce aggregate TCP and TLS startup time.

* Improve user-facing error experience around misconfigurations,
  by often assuming an HTTP protocol and yielding a more informative
  error.

* Overhauling metrics that we collect.

* Updating the authorization flow, laying groundwork for the
  UI to use the /authorize/user/task API (but not requiring it just
  yet).
    @jgraettinger
    jgraettinger committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    7b659c6 View commit details
    Browse the repository at this point in the history

  8. go.mod/Tiltfile: update for grpc-web and connector networking 

    Bring in Gazette updates for grpc-web gateways and net.Listener customization.

Deeply rework Tiltfile to remove data-plane-gateway,
by creating a self-signed TLS CA and Certificate that are used by the
broker and reactor (and may be used by other services if desired).

Use a naming strategy of `thing.flow.localhost`, because many libraries
like rustls don't accept wildcard certs of a toplevel `*.localhost`
but will happily accept `*.flow.localhost`.

Update for changes to --broker.allow-origin, --consumer.allow-origin,
and --flow.dashboard, as well as extracting explicit arguments into
environment variables.
    @jgraettinger
    jgraettinger committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    b1bbdde View commit details
    Browse the repository at this point in the history