Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Depreciate data-plane gateway authorization for shard and journal information #1264

Merged
merged 18 commits into from
Sep 24, 2024

Conversation

kiahna-tucker
Copy link
Member

@kiahna-tucker kiahna-tucker commented Sep 23, 2024

Issues

The issues directly below are advanced by this PR:
estuary/flow#1627

Changes

1627

  • Integrate the following control-plane APIs: /authorize/user/task and /authorize/user/collection.

  • Use the dynamic broker address and authorization token generated by the authorize collection, control-plane API to retrieve the journal data for data preview. Previously, the journal client was used to source this information.

  • Create a journal store to house journal state independent of the ops logs table.

  • Remove the warning that displayed on the Logs tab of the Details page in a local environment.

Tests

Manually tested

  • Validate that shard information can be fetched for display on the entity tables and the Details page of a given entity.

  • Validate that the authorize task, control-plane API is not called for collections.

  • Validate that journal information can be fetched for display in the ops logs table and in the data preview section of the Overview tab.

Automated tests

N/A

Playwright tests ran locally

  • Admin
  • Captures
  • Collections
  • HomePage
  • Login
  • Materialization

Screenshots

N/A

@kiahna-tucker kiahna-tucker added the change:planned This is a planned change label Sep 23, 2024
@kiahna-tucker kiahna-tucker marked this pull request as ready for review September 24, 2024 00:02
@kiahna-tucker kiahna-tucker requested a review from a team as a code owner September 24, 2024 00:02
@@ -29,6 +29,7 @@ const baseColumns = [
'spec_type',
'updated_at',
'last_pub_id',
'shard_template_id',
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎉 🎉 🎉 🎉 🎉 🎉

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Future change (cause my branch is not done yet) but I'll probably pull this filter into the entity-utils file.

Comment on lines +47 to +48
brokerAddress &&
brokerToken
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I really wish typescript was smart enough to not need this.

Comment on lines +68 to 71
dataPlaneListResponse.result.journals &&
dataPlaneListResponse.result.journals.length > 0
? dataPlaneListResponse.result.journals
: [],
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we know that the dataPlaneListResponse.result.journals is always an array then I think we could just always return it. Cause even if it is an empty array then that is basically what we're defaulting here.

) {
await refreshAuthToken(
session.access_token,
collectionName,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is fine for now... but we probably don't want to use the collectionName from the outer scope. Rather I think we can read it from the swr config that should be getting passed in to the onError.

taskSelector,
'ShardsList'
);
const shardPromises = catalogNames.map(async (name) => {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we need to limit this and make sure there is some limits of how many will be called all at once.

Copy link
Member

@travjenkins travjenkins left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lgtm

@kiahna-tucker kiahna-tucker merged commit 2da52bf into main Sep 24, 2024
3 checks passed
@kiahna-tucker kiahna-tucker deleted the kiahna-tucker/depreciate-data-plane-gateway branch September 24, 2024 18:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
change:planned This is a planned change
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants