Skip to content

Commit

Permalink
chore: align awss3 env variables to the AWS KMS
Browse files Browse the repository at this point in the history
This commit aligns the AWS S3 Storage services to the AWS KMS
The user will not require to set two sets of different env variables.

It makes easier to combine the two AWS services during the deployment.

Example:

```
      - RSTUF_STORAGE_BACKEND=AWSS3
      - RSTUF_AWS_STORAGE_BUCKET=tuf-metadata
      - RSTUF_AWS_ACCESS_KEY_ID=keyid
      - RSTUF_AWS_SECRET_ACCESS_KEY=ACCESS_KEY
      - RSTUF_AWS_DEFAULT_REGION=us-east-1
      - RSTUF_AWS_ENDPOINT_URL=http://localstack:4566
```

This commit also include updates in the Docker_README

Signed-off-by: Kairo Araujo <[email protected]>
  • Loading branch information
kairoaraujo committed Mar 7, 2024
1 parent c4b26c0 commit c3ff903
Show file tree
Hide file tree
Showing 3 changed files with 40 additions and 47 deletions.
31 changes: 12 additions & 19 deletions docs/source/guide/Docker_README.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,6 @@ Here are some things you need to know:
[Secure Systems Library](https://github.com/secure-systems-lab/securesystemslib).
If you do not have a key we suggest you use the [RSTUF CLI tool to generate the key](https://repository-service-tuf.readthedocs.io/en/latest/guide/repository-service-tuf-cli/index.html).
* This key must be the same one used during the [RSTUF CLI ceremony](https://repository-service-tuf.readthedocs.io/en/latest/guide/repository-service-tuf-cli/index.html#ceremony-ceremony).
* This key must be available to RSTUF Worker using the `RSTUF_KEYVAULT_BACKEND`.

For more information read the [Deployment documentation](https://repository-service-tuf.readthedocs.io/en/latest/guide/deployment/index.html).

Expand All @@ -41,10 +40,7 @@ For more information read the [Deployment documentation](https://repository-serv

```shell
docker run --env="RSTUF_STORAGE_BACKEND=LocalStorage" \
--env="RSTUF_LOCAL_STORAGE_BACKEND_PATH=storage" \
--env="RSTUF_KEYVAULT_BACKEND=LocalKeyVault" \
--env="RSTUF_LOCAL_KEYVAULT_PATH=keyvault" \
--env="RSTUF_LOCAL_KEYVAULT_KEYS=online.key,strongPass" \
--env="RSTUF_LOCAL_STORAGE_BACKEND_PATH=/metadata" \
--env="RSTUF_BROKER_SERVER=guest:guest@rabbitmq:5672" \
--env="RSTUF_REDIS_SERVER=redis://redis" \
--env="RSTUF_SQL_SERVER=postgresql://postgres:secret@postgres:5432" \
Expand Down Expand Up @@ -132,31 +128,31 @@ Available types:

##### `AWSS3` (AWS S3)

* (Required) ``RSTUF_AWSS3_STORAGE_BUCKET``
* (Required) ``RSTUF_AWS_STORAGE_BUCKET``

The name of the region associated with the S3.
The name of s3 bucket to use.

* (Required) ``RSTUF_AWSS3_STORAGE_ACCESS_KEY``
* (Required) ``RSTUF_AWS_ACCESS_KEY_ID``

The access key to use when creating the client session to the S3.

This environment variable supports container secrets when the ``/run/secrets``
volume is added to the path.
Example: `RSTUF_AWSS3_STORAGE_ACCESS_KEY=/run/secrets/S3_ACCESS_KEY`
Example: `RSTUF_AWS_ACCESS_KEY_ID=/run/secrets/S3_ACCESS_KEY`

* (Required) ``RSTUF_AWSS3_STORAGE_SECRET_KEY``
* (Required) ``RSTUF_AWS_SECRET_ACCESS_KEY``

The secret key to use when creating the client session to the S3.

This environment variable supports container secrets when the ``/run/secrets``
volume is added to the path.
Example: ``RSTUF_AWSS3_STORAGE_ACCESS_KEY=/run/secrets/S3_SECRET_KEY``
Example: ``RSTUF_AWS_SECRET_ACCESS_KEY=/run/secrets/S3_SECRET_KEY``

* (Optional) ``RSTUF_AWSS3_STORAGE_REGION``
* (Optional) ``RSTUF_AWS_DEFAULT_REGION``

The name of the region associated with the S3.

* (Optional) ``RSTUF_AWSS3_STORAGE_ENDPOINT_URL``
* (Optional) ``RSTUF_AWS_ENDPOINT_URL``

The complete URL to use for the constructed client. Normally, the
client automatically constructs the appropriate URL to use when
Expand All @@ -177,13 +173,12 @@ In most use cases, the timeout of 60.0 seconds is sufficient.

#### `RSTUF_KEYVAULT_BACKEND`

Select a supported type of Key Vault Service.
Available types:
Select a supported type of Key Vault Service.

* `LocalKeyVault` (container volume)

**_NOTE:_** You can start the worker
service without a keyvault backend, but you need to configure one before the
service without a keyvault backend, but you need to configure one before the
[bootstrap ceremony](https://repository-service-tuf.readthedocs.io/en/latest/guide/repository-service-tuf-cli/index.html#ceremony-ceremony).

##### `LocalKeyVault` (container volume)
Expand Down Expand Up @@ -232,7 +227,7 @@ service without a keyvault backend, but you need to configure one before the

Example: ``RSTUF_LOCAL_KEYVAULT_KEYS=/run/secrets/ONLINE_KEY_1:/run/secrets/ONLINE_KEY_2``

#### (Optional, *experimental*) `RSTUF_ONLINE_KEY_DIR`
#### (Optional) `RSTUF_ONLINE_KEY_DIR`

Directory path for online signing key file. Expected file format is unencrypted PKCS8/PEM.

Expand All @@ -246,8 +241,6 @@ Example:
- RSTUF worker expects related private key under `/run/secrets/<file name>`




#### (Optional) `RSTUF_WORKER_ID`

Custom Worker ID. Default: `hostname` (Container hostname)
Expand Down
20 changes: 10 additions & 10 deletions repository_service_tuf_worker/services/storage/awss3.py
Original file line number Diff line number Diff line change
Expand Up @@ -40,10 +40,10 @@ def __init__(

@classmethod
def configure(cls, settings: Dynaconf) -> "AWSS3":
access_key = parse_if_secret(settings.AWSS3_STORAGE_ACCESS_KEY)
secret_access_key = parse_if_secret(settings.AWSS3_STORAGE_SECRET_KEY)
region = settings.get("AWSS3_STORAGE_REGION")
endpoint = settings.get("AWSS3_STORAGE_ENDPOINT_URL")
access_key = parse_if_secret(settings.AWS_ACCESS_KEY_ID)
secret_access_key = parse_if_secret(settings.AWS_SECRET_ACCESS_KEY)
region = settings.get("AWS_DEFAULT_REGION")
endpoint = settings.get("AWS_ENDPOINT_URL")

s3_session = boto3.Session(
aws_access_key_id=access_key,
Expand All @@ -58,7 +58,7 @@ def configure(cls, settings: Dynaconf) -> "AWSS3":
endpoint_url=endpoint,
)
buckets = [bucket.name for bucket in s3_resource.buckets.all()]
bucket_name = settings.AWSS3_STORAGE_BUCKET
bucket_name = settings.AWS_STORAGE_BUCKET
if bucket_name not in buckets:
raise ValueError(f"Bucket '{bucket_name}' not found.")

Expand All @@ -78,23 +78,23 @@ def configure(cls, settings: Dynaconf) -> "AWSS3":
def settings(cls) -> List[ServiceSettings]:
return [
ServiceSettings(
names=["AWSS3_STORAGE_BUCKET"],
names=["AWS_STORAGE_BUCKET"],
required=True,
),
ServiceSettings(
names=["AWSS3_STORAGE_ACCESS_KEY"],
names=["AWS_ACCESS_KEY_ID"],
required=True,
),
ServiceSettings(
names=["AWSS3_STORAGE_SECRET_KEY"],
names=["AWS_SECRET_ACCESS_KEY"],
required=True,
),
ServiceSettings(
names=["AWSS3_STORAGE_REGION"],
names=["AWS_DEFAULT_REGION"],
required=False,
),
ServiceSettings(
names=["AWSS3_STORAGE_ENDPOINT_URL"],
names=["AWS_ENDPOINT_URL"],
required=False,
),
]
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -46,9 +46,9 @@ def test_full_init(self, mocked_boto3):
def test_configure(self, mocked_boto3):
test_settings = pretend.stub(
get=pretend.call_recorder(lambda *a: None),
AWSS3_STORAGE_BUCKET="bucket",
AWSS3_STORAGE_ACCESS_KEY="access_key",
AWSS3_STORAGE_SECRET_KEY="secret_key",
AWS_STORAGE_BUCKET="bucket",
AWS_ACCESS_KEY_ID="access_key",
AWS_SECRET_ACCESS_KEY="secret_key",
)

service = awss3.AWSS3.configure(test_settings)
Expand Down Expand Up @@ -91,15 +91,15 @@ def test_configure(self, mocked_boto3):

def test_configure_bucket_not_found(self, mocked_boto3):
def _fake_get(key: str) -> Optional[str]:
if key == "AWSS3_STORAGE_REGION":
if key == "AWS_DEFAULT_REGION":
return "region"
return None

test_settings = pretend.stub(
get=pretend.call_recorder(lambda a: _fake_get(a)),
AWSS3_STORAGE_BUCKET="nonexistent-bucket",
AWSS3_STORAGE_ACCESS_KEY="access_key",
AWSS3_STORAGE_SECRET_KEY="secret_key",
AWS_STORAGE_BUCKET="nonexistent-bucket",
AWS_ACCESS_KEY_ID="access_key",
AWS_SECRET_ACCESS_KEY="secret_key",
)

service = None
Expand Down Expand Up @@ -137,23 +137,23 @@ def test_settings(self, mocked_boto3):

assert service_settings == [
awss3.ServiceSettings(
names=["AWSS3_STORAGE_BUCKET"],
names=["AWS_STORAGE_BUCKET"],
required=True,
),
awss3.ServiceSettings(
names=["AWSS3_STORAGE_ACCESS_KEY"],
names=["AWS_ACCESS_KEY_ID"],
required=True,
),
awss3.ServiceSettings(
names=["AWSS3_STORAGE_SECRET_KEY"],
names=["AWS_SECRET_ACCESS_KEY"],
required=True,
),
awss3.ServiceSettings(
names=["AWSS3_STORAGE_REGION"],
names=["AWS_DEFAULT_REGION"],
required=False,
),
awss3.ServiceSettings(
names=["AWSS3_STORAGE_ENDPOINT_URL"],
names=["AWS_ENDPOINT_URL"],
required=False,
),
]
Expand Down Expand Up @@ -390,9 +390,9 @@ def test_get_DeserializationError(self, mocked_boto3):
def test_put(self, mocked_boto3):
test_settings = pretend.stub(
get=pretend.call_recorder(lambda *a: None),
AWSS3_STORAGE_BUCKET="bucket",
AWSS3_STORAGE_ACCESS_KEY="access_key",
AWSS3_STORAGE_SECRET_KEY="secret_key",
AWS_STORAGE_BUCKET="bucket",
AWS_ACCESS_KEY_ID="access_key",
AWS_SECRET_ACCESS_KEY="secret_key",
)

service = awss3.AWSS3.configure(test_settings)
Expand All @@ -410,9 +410,9 @@ def test_put(self, mocked_boto3):
def test_put_ClientErro(self, mocked_boto3):
test_settings = pretend.stub(
get=pretend.call_recorder(lambda *a: None),
AWSS3_STORAGE_BUCKET="bucket",
AWSS3_STORAGE_ACCESS_KEY="access_key",
AWSS3_STORAGE_SECRET_KEY="secret_key",
AWS_STORAGE_BUCKET="bucket",
AWS_ACCESS_KEY_ID="access_key",
AWS_SECRET_ACCESS_KEY="secret_key",
)
service = awss3.AWSS3.configure(test_settings)

Expand Down

0 comments on commit c3ff903

Please sign in to comment.