Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

source-http-ingest: support path parameters #2198

Merged
merged 1 commit into from
Dec 12, 2024
Merged

Conversation

psFried
Copy link
Member

@psFried psFried commented Dec 12, 2024

Description:

Introduces support for capturing path parameters from incoming requests. The paths in the endpoint config are now allowed to specify parameters using the OpenAPI syntax. For example, the path
/vendors/{vendorId}/products/{productId} can now be used to extract the vendorId and productId, which will be added to a /_meta/pathParams JSON object. The OpenAPI syntax was chosen because it's well-known and documented, and because it's simple and limited. This makes it easy to parse and less likely to result in surprising behavior. The syntax is documented at: https://swagger.io/docs/specification/v3_0/paths-and-operations/#path-templating

The /_meta/reqPath property was also added in order to record the configured path at which the request was handled.

Any path parameters are added as required properties in the discovered collection writeSchema. This allows them to be used as collection keys just by editing the collection in the UI during capture creation.

Workflow steps:

  • Enter a path in the endpoint config like /{paramA}/{paramB}
  • Send a request to /foo/bar
  • The captured document will include:
{
  "_meta": {
    "pathParams": {"paramA": "foo", "paramB": "bar"}, 
    "reqPath": "/{paramA}/{paramB}"
  }
}

Documentation links affected:

I'll put up a separate PR to update: https://docs.estuary.dev/reference/Connectors/capture-connectors/http-ingest/

Notes for reviewers:

This feature was suggested by a user in Slack, who would otherwise need to configure many different endpoint paths.


This change is Reviewable

Introduces support for capturing path parameters from incoming requests.  The
paths in the endpoint config are now allowed to specify parameters using the
OpenAPI syntax.  For example, the path
`/vendors/{vendorId}/products/{productId}` can now be used to extract the
`vendorId` and `productId`, which will be added to a `/_meta/pathParams` JSON
object. The OpenAPI syntax was chosen because it's well-known and documented,
and because it's simple and limited. This makes it easy to parse and less
likely to result in surprising behavior. The syntax is documented at:
https://swagger.io/docs/specification/v3_0/paths-and-operations/#path-templating

The `/_meta/reqPath` property was also added in order to record the configured
path at which the request was handled.
Copy link
Member

@williamhbaker williamhbaker left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@psFried psFried merged commit 68e82ac into main Dec 12, 2024
52 of 53 checks passed
@psFried psFried deleted the phil/webhook-path-captures branch December 12, 2024 15:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants