source-http-ingest: support path parameters #2198
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description:
Introduces support for capturing path parameters from incoming requests. The paths in the endpoint config are now allowed to specify parameters using the OpenAPI syntax. For example, the path
/vendors/{vendorId}/products/{productId}
can now be used to extract thevendorId
andproductId
, which will be added to a/_meta/pathParams
JSON object. The OpenAPI syntax was chosen because it's well-known and documented, and because it's simple and limited. This makes it easy to parse and less likely to result in surprising behavior. The syntax is documented at: https://swagger.io/docs/specification/v3_0/paths-and-operations/#path-templatingThe
/_meta/reqPath
property was also added in order to record the configured path at which the request was handled.Any path parameters are added as required properties in the discovered collection writeSchema. This allows them to be used as collection keys just by editing the collection in the UI during capture creation.
Workflow steps:
/{paramA}/{paramB}
/foo/bar
Documentation links affected:
I'll put up a separate PR to update: https://docs.estuary.dev/reference/Connectors/capture-connectors/http-ingest/
Notes for reviewers:
This feature was suggested by a user in Slack, who would otherwise need to configure many different endpoint paths.
This change is