Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Upload folder to cloud storage #38

Merged
merged 17 commits into from
Jan 4, 2023
Merged

Upload folder to cloud storage #38

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
17 commits
Select commit Hold shift + click to select a range
5eb2c46
created a common function to upload folder to cloud storage and extra…
Shofiya2003 Jan 2, 2023
5217c8f
completed implementing and testing the independent function to upload…
Shofiya2003 Jan 2, 2023
bd6c86d
implemented independent function to upload files to google cloud prov…
Shofiya2003 Jan 2, 2023
a09407d
created a struct for constants required to upload files to azure stor…
Shofiya2003 Jan 2, 2023
40ccdc6
created an interface with upload method to be implemented by all the …
Shofiya2003 Jan 2, 2023
9709dfe
packages imported
Shofiya2003 Jan 2, 2023
a563084
refactored the method to upload files on aws
Shofiya2003 Jan 2, 2023
a6e0dfe
added new environment variables to the template
Shofiya2003 Jan 2, 2023
1129c84
refactor:removed unnecessary comments
Shofiya2003 Jan 3, 2023
2720288
used bucketname for aws and gcp and accountname for azure from consta…
Shofiya2003 Jan 3, 2023
04ef31b
created common sessions for aws, client for gcp and credentials for a…
Shofiya2003 Jan 3, 2023
1113f1c
added constants
Shofiya2003 Jan 3, 2023
4ee3ba6
Merge branch 'dev' into upload_folder_to_cloud_storage
Shofiya2003 Jan 3, 2023
a221bb8
added value to constant GCP_BUCKET_NAME
Shofiya2003 Jan 3, 2023
aceb357
replaced 'log.Fatalln' with 'log.Println' to avoid termination of pro…
Shofiya2003 Jan 3, 2023
7da2c9e
Merge branch 'upload_folder_to_cloud_storage' of https://github.com/S…
Shofiya2003 Jan 3, 2023
29fedbf
removed use of defer for closing the file, instead closed it at the e…
Shofiya2003 Jan 4, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 5 additions & 1 deletion .env-template
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,2 +1,6 @@
PORT=8080
KAFKA_URI=kafka_uri
KAFKA_URI=kafka_uri
AWS_ACCESS_KEY_ID=aws_access_key_id
AWS_SECRET_ACCESS_KEY=aws_secret_access_key
GCP_CREDENTIALS=gcp_credentials (the path of the json file)
AZURE_ACCESS_KEY=azure_access_key
5 changes: 5 additions & 0 deletions constants/constants.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,8 +9,13 @@ const DEFAULT_PORT = "8080"

const DOWNLOAD_FILE_PATH_PREFIX = "downloads"
const DOWNLOAD_FOLDER_PERM = 0666

const S3_BUCKET_NAME = "zstream-bucket"
const S3_REGION = "us-east-1"
const AWS_ENDPOINT = "http://localhost:4566"
const PRESIGNED_URL_EXPIRATION = 60 * time.Minute
const OUTPUT_FILE_PATH_PREFIX = "output"
const GCP_BUCKET_NAME = "zstream-bucket"
const GCP_PROJECT_ID = ""
const AZURE_ACCOUNT_NAME = ""
const AZURE_ENDPOINT = ""
21 changes: 21 additions & 0 deletions go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,16 +12,30 @@ require (
)

require (
cloud.google.com/go v0.105.0 // indirect
cloud.google.com/go/compute v1.12.1 // indirect
cloud.google.com/go/compute/metadata v0.2.1 // indirect
cloud.google.com/go/iam v0.7.0 // indirect
cloud.google.com/go/storage v1.28.1 // indirect
github.com/Azure/azure-pipeline-go v0.2.3 // indirect
github.com/Azure/azure-storage-blob-go v0.15.0 // indirect
github.com/davecgh/go-spew v1.1.1 // indirect
github.com/gin-contrib/sse v0.1.0 // indirect
github.com/go-playground/locales v0.14.0 // indirect
github.com/go-playground/universal-translator v0.18.0 // indirect
github.com/go-playground/validator/v10 v10.11.1 // indirect
github.com/goccy/go-json v0.9.11 // indirect
github.com/golang/groupcache v0.0.0-20200121045136-8c9f03a8e57e // indirect
github.com/golang/protobuf v1.5.2 // indirect
github.com/google/go-cmp v0.5.9 // indirect
github.com/google/uuid v1.3.0 // indirect
github.com/googleapis/enterprise-certificate-proxy v0.2.0 // indirect
github.com/googleapis/gax-go/v2 v2.7.0 // indirect
github.com/jmespath/go-jmespath v0.4.0 // indirect
github.com/json-iterator/go v1.1.12 // indirect
github.com/klauspost/compress v1.15.9 // indirect
github.com/leodido/go-urn v1.2.1 // indirect
github.com/mattn/go-ieproxy v0.0.1 // indirect
github.com/mattn/go-isatty v0.0.16 // indirect
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect
github.com/modern-go/reflect2 v1.0.2 // indirect
Expand All @@ -30,10 +44,17 @@ require (
github.com/pmezard/go-difflib v1.0.0 // indirect
github.com/u2takey/go-utils v0.3.1 // indirect
github.com/ugorji/go/codec v1.2.7 // indirect
go.opencensus.io v0.24.0 // indirect
golang.org/x/crypto v0.0.0-20220622213112-05595931fe9d // indirect
golang.org/x/net v0.4.0 // indirect
golang.org/x/oauth2 v0.0.0-20221014153046-6fdb5e3db783 // indirect
golang.org/x/sys v0.3.0 // indirect
golang.org/x/text v0.5.0 // indirect
golang.org/x/xerrors v0.0.0-20220907171357-04be3eba64a2 // indirect
google.golang.org/api v0.103.0 // indirect
google.golang.org/appengine v1.6.7 // indirect
google.golang.org/genproto v0.0.0-20221201164419-0e50fba7f41c // indirect
google.golang.org/grpc v1.50.1 // indirect
google.golang.org/protobuf v1.28.1 // indirect
gopkg.in/yaml.v2 v2.4.0 // indirect
gopkg.in/yaml.v3 v3.0.1 // indirect
Expand Down
Loading