remove data-plane-gateway implementation and make it a simple proxy

[jgraettinger]

Since data-plane-gateway was written:

• The connector networking frontend was moved into reactors and significantly improved.

• gRPC Web REST handlers were implemented in reactors and gazette.

• reactors and gazette implemented first-class fine grain authorizations.

Remove the historical implementations of these features, instead updating to the implementations used by reactors and brokers today.

Remove authorization checks, and just verify and pass-through an authorization header.

Issue #estuary/flow/issues/1627

[jgraettinger]

I've been testing this with a local stack, adding to Tiltfile as:

local_resource(
    'data-plane-gateway',
    serve_dir='%s/data-plane-gateway' % REPO_BASE,
    serve_cmd='go run %s/data-plane-gateway serve \
    --broker.address https://gazette.flow.localhost:8080 \
    --consumer.address https://reactor.flow.localhost:9000 \
    --flow.control-api http://agent.flow.localhost:8675 \
    --flow.dashboard   http://localhost:3000 \
    --flow.data-plane-fqdn local-cluster.dp.estuary-data.com \
    --gateway.allow-origin http://localhost:3000 \
    --gateway.host gateway.flow.localhost \
    --gateway.port 28318 \
    --log.level info \
    ' % (REPO_BASE),
    serve_env={
        "BROKER_TRUSTED_CA_FILE": CA_CERT_PATH,
        "CONSUMER_AUTH_KEYS": AUTH_KEYS,
        "CONSUMER_TRUSTED_CA_FILE": CA_CERT_PATH,
        "GATEWAY_SERVER_CERT_FILE": TLS_CERT_PATH,
        "GATEWAY_SERVER_CERT_KEY_FILE": TLS_KEY_PATH,
    },
    links='https://gateway.flow.localhost:28318/debug/pprof',
    resource_deps=['gazette', 'reactor'],
    readiness_probe=probe(
        initial_delay_secs=5,
        http_get=http_get_action(port=9000, path='/debug/ready', scheme='https')
    ),
)

Things I've tried:

• flowctl --profile local logs --task some/task
• flowctl-go with BROKER_ADDRESS and CONSUMER_ADDRESS set to the local gateway address.
• Creating a source-http-ingest capture, source-hello, and materialize-sqlite materialization.
• Updating the data_plane columns for broker and reactor address to actually be the gateway.
• Verified that UI still functions correctly
  • I see gRPC web requests are directed to the gateway for shard status, log reads, etc.
• With the data-plane row updated, the UI shows gateway based addresses for connectors
  • These load and function correctly, going through the gateway network frontend.
• I also see the reactor issuing List RPCs, as well as some Reads and Appends, to the gateway without issue.

[psFried]

LGTM, nice to see so much of this get deleted!

[psFried]

What does this change do? Why is 65534 better than nonroot?

[jgraettinger]

doesn't exist in ubuntu:24.04

65534 is nobody (which curiously also didn't trivially work, but i didn't look into it further)