feat!: shared vpc updates, new database and endpoint configuration, a…

…nd semantic releases (#227)

.github/workflows/develop.yml

@@ -113,5 +113,23 @@ jobs:
       - name: Deploy 
         run: |
           echo $STAGE
-          cdk deploy veda-backend-uah-dev
-          
+          cdk deploy --require-approval never
+
+  pre-release:
+    needs: [deploy]
+    runs-on: ubuntu-latest
+    concurrency: release
+    permissions:
+      id-token: write
+      contents: write
+
+    steps:
+    - uses: actions/checkout@v3
+      with:
+        fetch-depth: 0
+    - name: Python Semantic Release
+      uses: python-semantic-release/python-semantic-release@master
+      with:
+        changelog: "false"
+        vcs-release: "true"
+        github_token: ${{ secrets.GITHUB_TOKEN }}

.github/workflows/main.yml

@@ -113,5 +113,24 @@ jobs:
       - name: Deploy 
         run: |
           echo $STAGE
-          cdk deploy veda-backend-uah-staging
+          cdk deploy --require-approval never
+
+  release:
+    needs: [deploy]
+    runs-on: ubuntu-latest
+    concurrency: release
+    permissions:
+      id-token: write
+      contents: write
+
+    steps:
+    - uses: actions/checkout@v3
+      with:
+        fetch-depth: 0
+    - name: Python Semantic Release
+      uses: python-semantic-release/python-semantic-release@master
+      with:
+        changelog: "false"
+        vcs-release: "true"
+        github_token: ${{ secrets.GITHUB_TOKEN }}
 

.github/workflows/pr.yml

@@ -25,8 +25,20 @@ jobs:
       - name: Run pre-commit
         run: pre-commit run --all-files
 
+  lint-conventional-pr:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v2
+      - uses: CondeNast/[email protected]
+        env:
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+        with:
+          # if the PR contains a single commit, fail if the commit message and the PR title do not match
+          commitTitleMatch: "true"
+          ignoreCommits: "true"
+
   test:
-    needs: [lint]
+    needs: [lint, lint-conventional-pr]
     runs-on: ubuntu-latest
     steps:
       - uses: actions/checkout@v3

CONTRIBUTING.md

@@ -0,0 +1,58 @@
+# Contributing to veda-backend
+
+## Collaboration
+
+This project generally follows the [git-flow branching model](https://nvie.com/posts/a-successful-git-branching-model/).
+
+### Branching
+
+- All features, fixes, and improvements are committed in branches based on the **`develop`** protected branch.
+
+- As soon as manual smoke testing has been executed in the development environment, a new pull request is used to broadcast and get wider approval from maintainers for the new work before merging **`develop`** into the protected **`main`** branch.
+
+### Pull request conventions
+
+- Pull requests should explain **what** is changed, **why**, and **how the change can be tested**.
+
+- Pull request *titles* must adhere to the [conventional commits specification]([#conventional-commits](https://www.conventionalcommits.org/en/v1.0.0/)). This is enforced using the  [conventional-pull-request-action](https://github.com/CondeNast/conventional-pull-request-action).  
+
+- Successful automated pull request linting, testing, and deployment checks are required.
+
+- Deleting the feature branch after merging is recommended but not enforced.
+
+
+## Conventional commits and semantic versioning
+
+[Conventional Commits v1.0.0 Specification](https://www.conventionalcommits.org/en/v1.0.0/) compliant commit messages and pull request titles allow us to systematically generate the next [Semantic Versioning 2.0.0](https://semver.org/#semantic-versioning-200) MAJOR.MINOR.PATCH version. Conventional commits are strongly recommended for all changes, but are only required for pull request titles (and for the single commit in a pull request that only includes one commit). We use [python-semantic-release](https://github.com/python-semantic-release/python-semantic-release) to infer version steps from conventional commit messages.
+
+### Conventional commits and MAJOR.MINOR.PATCH semantic version
+
+This project currently implements python-semantic-release defaults in [pyproject.toml](pyproject.toml) where custom versioning strategy [configuration](https://python-semantic-release.readthedocs.io/en/latest/configuration.html) can be added. Local installation of python-semantic-release is [optional](https://python-semantic-release.readthedocs.io/en/latest/#installation).
+
+- Major version increments are required when breaking changes are introduced. Breaking changes are indicated with a **`!`** in a conventional commit message such as `feat(database)!:` ([example](https://www.conventionalcommits.org/en/v1.0.0/#commit-message-with--to-draw-attention-to-breaking-change)).
+
+- Minor version increments are applied on feature changes, these commit messages are prefixed with `feat:` or `feat(scope):`.
+
+- Patch version increments are applied for fixes and performance tuning by using the `fix:` and `perf:` commit prefixes with or without a defined `(scope)`.
+
+- Chores, documentation, and other changes that do not cause version changes including `"chore"`, `"ci"`, `"docs"`, `"style"`, `"refactor"`, and `"test"` are configured in `pyproject.toml`.
+
+### Automated releases
+
+Release branch groups are configured in [pyproject.toml](pyproject.toml). Merges in to the develop branch generate pre release candidates. When the develop branch is merged into main, new releases are generated by dropping the `rc.<increment>` from the release candidate version. Release changes are generated from conventional commit messages.
+
+> **Note:** Release versions are managed with git tags only to avoid managing an admin token for the release action.
+
+## Tips and tools
+
+### [Commitizen](https://github.com/commitizen/cz-cli) git prompter
+
+A command line tool with prompts for formatted conventional commit messages.
+
+### Fix missed conventional commit messages
+
+Empty commit messages are an easy way to add to the conventional commit history used for semantic versioning. `git commit --allow-empty -m 'feat(api)!: introduced breaking path change in xyz endpoint'`
+
+### Squased merges (if you must)
+
+If it is necessary to squash commit history on merging, make sure that the merge message is an appropriate conventional commit message to ensure that automated release versioning will be applied properly. The project is configured to default the squash commit message to the pull request title.

README.md

@@ -43,7 +43,17 @@ To retrieve the variables for a stage that has been previously deployed, the sec
 | `VEDA_DB_PGSTAC_VERSION` | **REQUIRED** version of PgStac database, i.e. 0.7.6 |
 | `VEDA_DB_SCHEMA_VERSION` | **REQUIRED** The version of the custom veda-backend schema, i.e. 0.1.1 |
 | `VEDA_DB_SNAPSHOT_ID` | **Once used always REQUIRED** Optional RDS snapshot identifier to initialize RDS from a snapshot |
-> **Note** See [Advanced Configuration](docs/advanced_configuration.md) for details about custom configuration options.
+
+### Advanced configuration
+The constructs and applications in this project are configured using pydantic. The settings are defined in config.py files stored alongside the associated construct or application--for example the settings for the RDS PostgreSQL construct are defined in database/infrastructure/config.py. For custom configuration, use environment variables to override the pydantic defaults.
+
+| Construct | Env Prefix | Configuration |
+| --- | --- | --- |
+| Database | `VEDA_DB` | [database/infrastructure/config.py](database/infrastructure/config.py) |
+| Domain | `VEDA_DOMAIN` | [domain/infrastructure/config.py](domain/infrastructure/config.py) |
+| Network | `N/A` | [network/infrastructure/config.py](network/infrastructure/config.py) |
+| Raster API (TiTiler) | `VEDA_RASTER` | [raster_api/infrastructure/config.py](raster_-_api/infrastructure/config.py) |
+| STAC API | `VEDA_STAC` | [stac_api/infrastructure/config.py](stac_api/infrastructure/config.py) |
 
 ### Deploying to the cloud
 
@@ -87,8 +97,9 @@ cdk deploy
 If this is a development stack that is safe to delete, you can delete the stack in CloudFormation console or via `cdk destroy`, however, the additional manual steps were required to completely delete the stack resources:
 
 1. You will need to disable deletion protection of the RDS database and delete the database.
-2. Detach the Internet Gateway (IGW) from the VPC and delete it.
-3. If this stack created a new VPC, delete the VPC (this should delete a subnet and security group too).
+2. Identify and delete the RDS subnet group associated with the RDS database you just deleted (it will not be automatically removed because of the RDS deletion protection in place when the group was created).
+3. If this stack created a new VPC, detach the Internet Gateway (IGW) from the VPC and delete it.
+4. If this stack created a new VPC, delete the VPC (this should delete a subnet and security group too).
 
 ## Custom deployments
 

app.py

@@ -1,13 +1,14 @@
 #!/usr/bin/env python3
 """ CDK Configuration for the veda-backend stack."""
 
-from aws_cdk import App, Stack, Tags, aws_iam
+from aws_cdk import App, Aspects, Stack, Tags, aws_iam
 from constructs import Construct
 
 from config import veda_app_settings
 from database.infrastructure.construct import RdsConstruct
 from domain.infrastructure.construct import DomainConstruct
 from network.infrastructure.construct import VpcConstruct
+from permissions_boundary.infrastructure.construct import PermissionsBoundaryAspect
 from raster_api.infrastructure.construct import RasterApiLambdaConstruct
 from stac_api.infrastructure.construct import StacApiLambdaConstruct
 
@@ -22,12 +23,13 @@ def __init__(self, scope: Construct, construct_id: str, **kwargs) -> None:
         super().__init__(scope, construct_id, **kwargs)
 
         if veda_app_settings.permissions_boundary_policy_name:
-            permission_boundary_policy = aws_iam.Policy.from_policy_name(
+            permissions_boundary_policy = aws_iam.Policy.from_policy_name(
                 self,
-                "permission-boundary",
+                "permissions-boundary",
                 veda_app_settings.permissions_boundary_policy_name,
             )
-            aws_iam.PermissionsBoundary.of(self).apply(permission_boundary_policy)
+            aws_iam.PermissionsBoundary.of(self).apply(permissions_boundary_policy)
+            Aspects.of(self).add(PermissionsBoundaryAspect(permissions_boundary_policy))
 
 
 veda_stack = VedaStack(

database/infrastructure/config.py

@@ -1,6 +1,7 @@
 """Veda-backend database construct configuration."""
 from typing import Optional
 
+from aws_cdk import aws_ec2, aws_rds
 from pydantic import BaseSettings, Field
 
 
@@ -57,6 +58,38 @@ class vedaDBSettings(BaseSettings):
         False,
         description="Boolean if the RDS should be accessed through a proxy",
     )
+    rds_instance_class: Optional[str] = Field(
+        aws_ec2.InstanceClass.BURSTABLE3.value,
+        description=(
+            "The instance class of the RDS instance "
+            "https://docs.aws.amazon.com/cdk/api/v2/python/aws_cdk.aws_ec2/InstanceClass.html"
+        ),
+    )
+    rds_instance_size: Optional[str] = Field(
+        aws_ec2.InstanceSize.SMALL.value,
+        description=(
+            "The size of the RDS instance "
+            "https://docs.aws.amazon.com/cdk/api/v2/python/aws_cdk.aws_ec2/InstanceSize.html"
+        ),
+    )
+    rds_engine_full_version: Optional[str] = Field(
+        aws_rds.PostgresEngineVersion.VER_14.postgres_full_version,
+        description=(
+            "The version of the RDS Postgres engine "
+            "https://docs.aws.amazon.com/cdk/api/v2/python/aws_cdk.aws_rds/PostgresEngineVersion.html"
+        ),
+    )
+    rds_engine_major_version: Optional[str] = Field(
+        aws_rds.PostgresEngineVersion.VER_14.postgres_major_version,
+        description=(
+            "The version of the RDS Postgres engine "
+            "https://docs.aws.amazon.com/cdk/api/v2/python/aws_cdk.aws_rds/PostgresEngineVersion.html"
+        ),
+    )
+    rds_encryption: Optional[bool] = Field(
+        False,
+        description="Boolean if the RDS should be encrypted",
+    )
 
     class Config:
         """model config."""

database/infrastructure/construct.py

@@ -137,8 +137,19 @@ def __init__(
 
         # Custom parameter group
         engine = aws_rds.DatabaseInstanceEngine.postgres(
-            version=aws_rds.PostgresEngineVersion.VER_14
+            version=aws_rds.PostgresEngineVersion.of(
+                veda_db_settings.rds_engine_full_version,
+                veda_db_settings.rds_engine_major_version,
+            )
+        )
+
+        # RDS Instance Type
+        rds_instance_type = aws_ec2.InstanceType.of(
+            aws_ec2.InstanceClass[veda_db_settings.rds_instance_class],
+            aws_ec2.InstanceSize[veda_db_settings.rds_instance_size],
         )
+
+        #  version=aws_rds.PostgresEngineVersion.postgres_major_version(veda_db_settings.rds_engine_version)
         parameter_group = aws_rds.ParameterGroup(
             self,
             "parameter-group",
@@ -150,48 +161,39 @@ def __init__(
             },
         )
 
+        # Database Configurations
+        database_config = {
+            "id": "rds",
+            "instance_identifier": f"{stack_name}-postgres",
+            "vpc": vpc,
+            "engine": engine,
+            "instance_type": rds_instance_type,
+            "vpc_subnets": aws_ec2.SubnetSelection(subnet_type=subnet_type),
+            "deletion_protection": True,
+            "removal_policy": RemovalPolicy.RETAIN,
+            "publicly_accessible": veda_db_settings.publicly_accessible,
+            "parameter_group": parameter_group,
+        }
+        if veda_db_settings.rds_encryption:
+            database_config["storage_encrypted"] = veda_db_settings.rds_encryption
+
         # Create a new database instance from snapshot if provided
         if veda_db_settings.snapshot_id:
             # For the database from snapshot we will need a new master secret
-            credentials = aws_rds.SnapshotCredentials.from_generated_secret(
+            snapshot_credentials = aws_rds.SnapshotCredentials.from_generated_secret(
                 username=veda_db_settings.admin_user
             )
 
             database = aws_rds.DatabaseInstanceFromSnapshot(
                 self,
-                id="rds",
                 snapshot_identifier=veda_db_settings.snapshot_id,
-                instance_identifier=f"{stack_name}-postgres",
-                vpc=vpc,
-                engine=engine,
-                instance_type=aws_ec2.InstanceType.of(
-                    aws_ec2.InstanceClass.BURSTABLE3, aws_ec2.InstanceSize.SMALL
-                ),
-                vpc_subnets=aws_ec2.SubnetSelection(subnet_type=subnet_type),
-                deletion_protection=True,
-                removal_policy=RemovalPolicy.RETAIN,
-                publicly_accessible=veda_db_settings.publicly_accessible,
-                credentials=credentials,
-                parameter_group=parameter_group,
+                credentials=snapshot_credentials,
+                **database_config,
             )
 
         # Or create/update RDS Resource
         else:
-            database = aws_rds.DatabaseInstance(
-                self,
-                id="rds",
-                instance_identifier=f"{stack_name}-postgres",
-                vpc=vpc,
-                engine=engine,
-                instance_type=aws_ec2.InstanceType.of(
-                    aws_ec2.InstanceClass.BURSTABLE3, aws_ec2.InstanceSize.SMALL
-                ),
-                vpc_subnets=aws_ec2.SubnetSelection(subnet_type=subnet_type),
-                deletion_protection=True,
-                removal_policy=RemovalPolicy.RETAIN,
-                publicly_accessible=veda_db_settings.publicly_accessible,
-                parameter_group=parameter_group,
-            )
+            database = aws_rds.DatabaseInstance(self, **database_config)
 
         hostname = database.instance_endpoint.hostname