Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add a basic support for parsing urls from virtual private cloud #49

Merged
merged 5 commits into from
Jan 24, 2024
Merged

Add a basic support for parsing urls from virtual private cloud #49

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
f1f907a
Add a basic support for parsing urls from virtual private cloud
Drvi Jan 24, 2024
b2d120f
Improve comment
Drvi Jan 24, 2024
b938a60
v1.6.2
Drvi Jan 24, 2024
04b88ab
PR feedback
Drvi Jan 24, 2024
5fd14eb
1.6 compat
Drvi Jan 24, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion Project.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
name = "CloudStore"
uuid = "3365d9ee-d53b-4a56-812d-5344d5b716d7"
authors = ["quinnj <[email protected]>"]
version = "1.6.1"
version = "1.6.2"

[deps]
Base64 = "2a0f44e3-6c83-55bd-87e4-b1978d98bd5f"
Expand Down
7 changes: 7 additions & 0 deletions src/parse.jl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@ const AWS_REGIONS = Set{String}([
"ap-southeast-2",
"ap-northeast-1",
"ca-central-1",
"ca-west-1",
"eu-central-1",
"eu-west-1",
"eu-west-2",
Expand All @@ -26,6 +27,7 @@ const AWS_REGIONS = Set{String}([
"eu-south-2",
"eu-north-1",
"eu-central-2",
"il-central-1",
"me-south-1",
"me-central-1",
"sa-east-1",
Expand Down Expand Up @@ -136,6 +138,11 @@ function parseAWSBucketRegionKey(url; parseLocal::Bool=false)
# https://bucket-name.s3.amazonaws.com
m = match(r"^https://(?<bucket>[^\.]+)\.s3(?<accelerate>-accelerate)?(?:\.(?<region>[^\.]+))?\.amazonaws\.com(?:/(?<key>.+))?$"i, url)
m !== nothing && return _validate_aws(true, !isnothing(m[:accelerate]), nothing, m[:bucket], m[:region], m[:key])

# https://bucket.vpce-1a2b3c4d-5e6f.s3.region-code.vpce.amazonaws.com/bucket-name/key-name
# https://bucket.vpce-1a2b3c4d-5e6f.s3.region-code.vpce.amazonaws.com/bucket-name
m = match(r"^https://bucket\.vpce[^\.]+\.s3\.(?<region>[^\.]+)\.vpce\.amazonaws\.com/(?<bucket>[^/]+)(?:/(?<key>.+))?$"i, url)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you link to docs for the schema?

Does this have to be https://bucket.vpce-... (with literal bucket) rather than https://<bucket-name-here>.vpce-...?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should be a literal bucket AFAIK. I'll link the docs 👍

m !== nothing && return _validate_aws(true, false, nothing, m[:bucket], m[:region], m[:key])
# https://s3.region-code.amazonaws.com/bucket-name/key-name
# https://s3.region-code.amazonaws.com/bucket-name
m = match(r"^https://s3(?:\.(?<region>[^\.]+))?\.amazonaws\.com/(?<bucket>[^/]+)(?:/(?<key>.+))?$"i, url)
Expand Down
3 changes: 3 additions & 0 deletions test/runtests.jl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -531,6 +531,9 @@ end
("S3://bucket-name", (true, false, nothing, "bucket-name", "", "")),
("HTtp://127.0.0.1:27181/bucket-name/key-name", (true, false, "HTtp://127.0.0.1:27181", "bucket-name", "", "key-name")),
("htTP://127.0.0.1:27181/bucket-name", (true, false, "htTP://127.0.0.1:27181", "bucket-name", "", "")),

("https://bucket.vpce-1a2b3c4d-5e6f.s3.us-west-2.vpce.amazonaws.com/bucket-name", (true, false, nothing, "bucket-name", "us-west-2", "")),
("https://bucket.vpce-1a2b3c4d-5e6f.s3.us-west-2.vpce.amazonaws.com/bucket-name/key-name", (true, false, nothing, "bucket-name", "us-west-2", "key-name")),
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

are there more things we should test other than these 2 happy paths?

e.g. what if the key-name has URI encoded special characters?

Are there AWS docs on what is / isn't valid?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, we use the same validation logic for all the URL styles, we apply it to the things we extract from the regex. I can add some cases that fail that validation.

]
for (url, parts) in s3
ok, accelerate, host, bucket, reg, key = CloudStore.parseAWSBucketRegionKey(url; parseLocal=true)
Expand Down
Loading