Merge pull request #701 from Worklytics/rc-v0.4.52

v0.4.52

.github/workflows/ci-terraform-examples-release.yaml

@@ -21,7 +21,7 @@ jobs:
           'examples/gcp-google-workspace',
           'examples/msft-365'
         ]
-        terraform_version: [ '~1.4.0', '~1.5.0', '~1.6.0' ]
+        terraform_version: [ '~1.4.0', '~1.5.0', '~1.6.0', '~1.7.0' ]
     uses: ./.github/workflows/ci-terraform-example.yaml
     with:
       terraform_version: ${{ matrix.terraform_version }}

.github/workflows/ci-terraform-examples.yaml

@@ -19,7 +19,7 @@ jobs:
                         'examples-dev/gcp',
                         'examples-dev/gcp-google-workspace',
         ]
-        terraform_version: [ '~1.4.0', '~1.5.0', '~1.6.0' ]
+        terraform_version: [ '~1.4.0', '~1.5.0', '~1.6.0', '~1.7.0' ]
     uses: ./.github/workflows/ci-terraform-example.yaml
     with:
       terraform_version: ${{ matrix.terraform_version }}

.github/workflows/ci-terraform-modules.yaml

@@ -11,7 +11,7 @@ jobs:
     runs-on: ubuntu-latest
     strategy:
       matrix:
-        terraform_version: [ '~1.4.0', '~1.5.0', '~1.6.0']
+        terraform_version: [ '~1.4.0', '~1.5.0', '~1.6.0', '~1.7.0']
     steps:
       - name: Check out code
         uses: actions/checkout@v4

CHANGELOG.md

@@ -20,6 +20,35 @@ Changes to be including in future/planned release notes will be added here.
         then wildcard policy to read shared also grants read of secrets across all connectors)
   - keys/salts per value kind (PII, item id, etc)
 
+## [0.4.52](https://github.com/Worklytics/psoxy/release/tag/v0.4.52)
+  - BREAKING: default behavior for sub-addressing aka "plus addressing" of emails has changed; the
+    proxy previously considered these canonically distinct. Now, the proxy will consider these
+    canonically equivalent. As we don't expect plus addressing to be used hris or directory data,
+    this should have little impact.  Changes will most likely be in a few edge cases, such as
+    emails or calendar invites sent to a sub-address - sender unlikely to be a subaddress, but
+    recipient could be.  In such cases, behavior prior to 0.4.52 would cause recipient to appear
+    as a distinct mailbox; from 0.4.52 onward, they will be considered the same mailbox; we expect
+    this to be behavior that is more in line with user expectations, so although technically
+    breaking, we're introducing it without a major version bump.
+  - there new option to enable less strict email canonicalization; we strongly recommend new
+    customers to enable it, although it is not enabled by default to avoid a breaking change. Set
+    `email_canonicalization` to `IGNORE_DOTS` to enable this feature.
+  - BREAKING for examples: default value fore `email_canonicalization` in our example repos has been
+    set to `IGNORE_DOTS`; if you've previously forked an example, this is not a breaking change. but
+    if you fork an example > 0.4.52 and are attempting to migrate a proxy deployment initially built
+    with modules or examples from < 0.4.52, you should explicitly add `email_canonicalization = "STRICT"`
+    in your `terraform.tfvars`
+  - GCP: Existing GCP functions are using *Container Registry* for building their internal docker image where the psoxy code is deployed. However, 
+    this is [deprecated since May 2023 and starting Feb 2024](https://cloud.google.com/container-registry/docs/deprecations/container-registry-deprecation) it 
+    is required that functions use *Artifact Registry* instead. All deployments made since this version will use *Artifact Registry*
+    default repository for storing all psoxy images. Any previous version before this version will work without any issue.
+
+## [0.4.51](https://github.com/Worklytics/psoxy/release/tag/v0.4.51)
+ - GCP: non-breaking, but noticeable in Terraform plan: `title` attribute of GCP Custom Project
+   roles created by our modules are changing to more closely follow conventions GCP uses for its
+   built-in roles; as well as prefixing them with your environment ID to group them together
+   alphabetically and differentiate in shared project.
+
 ## [0.4.50](https://github.com/Worklytics/psoxy/release/tag/v0.4.50)
   - `todos_as_local_files` properly respected now; if you had it as `false`, you may see some local
     files deleted on your next `terraform apply`.
@@ -282,4 +311,4 @@ Features:
 Upgrade Notes:
   - secret management has been refactored; you may see indications of some secrets being moved, or
     even destroyed and recreated. If you plan shows SALT or ENCRYPTION_KEY as being destroyed,
-    **DO NOT** apply the plan and contact Worklytics support for assistance.
+    **DO NOT** apply the plan and contact Worklytics support for assistance.

README.md

@@ -218,7 +218,7 @@ You will need all the following in your deployment environment (eg, your laptop)
 | [git](https://git-scm.com/)                     | 2.17+                  | `git --version`           |
 | [Maven](https://maven.apache.org/)              | 3.6+                   | `mvn -v`                 |
 | [Java JDK 11+](https://openjdk.org/install/) | 11, 17, 21 (see notes) | `mvn -v | grep Java` |
-| [Terraform](https://www.terraform.io/)          | 1.3.x, <= 1.6          | `terraform version`       |
+| [Terraform](https://www.terraform.io/)          | 1.3.x, <= 1.7.x          | `terraform version`       |
 
 NOTE: we will support Java versions for duration of official support windows, in particular the
 LTS versions. As of Nov 2023, we  still support java 11 but may end this at any time. Minor
@@ -233,6 +233,11 @@ tool, but we don't offer documentation or support in doing so.  Adapting one of
 
 NOTE: Refrain to use Terraform versions 1.4.x that are < v1.4.3. We've seen bugs.
 
+NOTE: As of proxy version 0.4.x, we're constraining our Terraform modules/examples to 1.3.x features,
+and will automate testing against latest release of each minor version from there forwards to 2.0,
+as we add support for each. Once we introduce testing of newer versions, we will update version
+constraint above and in our modules to reflect support for newer versions.
+
 Depending on your Cloud Host / Data Sources, you will need:
 
 | Condition                         | Tool                                                                                       | Version | Test Command     | Roles / Permissions (Examples, YMMV)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |

docs/development/terraform-versions.md

@@ -0,0 +1,36 @@
+# Terraform Version Compatibility
+
+We use CI to automate testing against *latest* version of each minor version of Terraform
+(eg 1.3.x, 1.4.x, etc that we support).
+
+Given that Hashicorp has changed Terraform licensing to ELv2; and that Linux foundation has forked
+Terraform from 1.5.x to [create OpenTofu](https://opentofu.org/blog/the-opentofu-fork-is-now-available/),
+we are targeting compatibility with 1.6.x feature set.
+
+Do NOT use any features from 1.3.x or later, as we wish to allow all our terraform modules/examples
+to work with Terraform back to 1.3.x, which is vintage Sept 2022.
+
+In particular, **features to NOT use**:
+  - `removed` block - introduced in 1.7
+  - `check` block - introduced in 1.5
+  - `import` block - introduced in 1.5, but not relevant in use case anyways.
+  - `plantimestamp` function - introduced in 1.5
+  - `strcontains` function - introduced in 1.5
+  - `terraform_data` resource - introduced in 1.4
+  - `gcs` backend `kms_encryption_key`, `storage_custom_endpoint` attributes - introduced in 1.4
+
+
+Features that we don't use as of March 2024, but likely safe:
+  - `terraform test` - introduced in 1.6; tests defined in separate .tftest.hcl files, so likely
+     defining such won't break compatibility with earlier versions or OpenTofu
+  - `quiet` attribute on `local-exec` - introduced in 1.4 ... might be safe if older versions that
+    don't know about it just ignore it.
+
+
+## Testing
+
+To test with a specific version of Terraform, we suggest [tfenv](https://github.com/tfutils/tfenv)
+tool. Install everything you need, and create `.terraform-version` file in the root of your
+terraform configuration with the desired version.
+
+

docs/guides/deployment-migration.md

@@ -45,6 +45,10 @@ What you MUST **copy**:
 - **value for `PSEUDONYMIZE_APP_IDS`.** This value, if set to `true` will have the proxy use a rule
   set that pseudonymizes identifiers issued by source applications themselves in some cases where
   these identifiers aren't inherently PII - but the association could be considered discoverable.
+- **value for `EMAIL_CANONICALIZATION`.** prior to v0.4.52, this default was in effect `STRICT`; so
+  if your original deployment was built on a version prior to this, you should explicitly set this
+  value to `STRICT` in your new configuration (likely `email_canonicalization` variable in terraform
+  modules)
 - any **custom sanitization rules** that you've set, either in your Terraform configuration or
   directly as the value of a `RULES` environment variable, SSM Parameter, or GCP Secret.
 - historical **sanitized files** for any bulk connectors, if you wish to continue to have this data

docs/troubleshooting.md

@@ -89,3 +89,27 @@ eg
 ```shell
 terraform import module.psoxy-msft-connector\[\"outlook-cal\"\].aws_lambda_function_url.lambda_url psoxy-outlook-cal
 ```
+
+### Unsupported Terraform versions
+
+Errors such as the following on `terraform plan`?
+```shell
+Module module.psoxy (from git::https://github.com/worklytics/psoxy//infra/modules/gcp-host?ref=v0.4.51) does not support Terraform version 1.8.1. To proceed, either choose another supported Terraform version or update
+│ this version constraint. Version constraints are normally set for good reason, so updating the constraint may lead to other errors or unexpected behavior.
+```
+
+The solution is to downgrade your Terraform version to one that's supported by our modules
+(>= 1.3.x, <= 1.7.x as of March 2024).
+
+_If you're running Terraform in cloud/CI environment,_ including Terraform Cloud, GitHub Actions, etc,
+you can likely explicitly set the desired Terraform version in your workspace settings / terraform
+setup action.
+
+_If you're running Terraform on your laptop or in a VM,_ use your package manager to downgrade or
+something like [`tfenv`](https://github.com/tfutils/tfenv) to concurrently use distinct Terraform
+versions on the machine. (set version <= 1.7.x in `.terraform-version` file in the root of your
+Terraform configuration for the proxy).
+
+
+
+

infra/examples-dev/aws-all/google-workspace.tf

@@ -8,7 +8,7 @@ provider "google" {
 
 module "worklytics_connectors_google_workspace" {
   source = "../../modules/worklytics-connectors-google-workspace"
-  # source = "git::https://github.com/worklytics/psoxy//infra/modules/worklytics-connectors-google-workspace?ref=v0.4.51"
+  # source = "git::https://github.com/worklytics/psoxy//infra/modules/worklytics-connectors-google-workspace?ref=v0.4.52"
 
   providers = {
     google = google.google_workspace

infra/examples-dev/aws-all/main.tf

@@ -1,4 +1,6 @@
 terraform {
+  required_version = ">= 1.3, < 1.8"
+
   required_providers {
     # for the infra that will host Psoxy instances
     aws = {
@@ -19,7 +21,7 @@ terraform {
 # general cases
 module "worklytics_connectors" {
   source = "../../modules/worklytics-connectors"
-  # source = "git::https://github.com/worklytics/psoxy//infra/modules/worklytics-connectors?ref=v0.4.51"
+  # source = "git::https://github.com/worklytics/psoxy//infra/modules/worklytics-connectors?ref=v0.4.52"
 
   enabled_connectors               = var.enabled_connectors
   jira_cloud_id                    = var.jira_cloud_id
@@ -97,7 +99,7 @@ locals {
 
 module "psoxy" {
   source = "../../modules/aws-host"
-  # source = "git::https://github.com/worklytics/psoxy//infra/modules/aws-host?ref=v0.4.51"
+  # source = "git::https://github.com/worklytics/psoxy//infra/modules/aws-host?ref=v0.4.52"
 
   environment_name                     = var.environment_name
   aws_account_id                       = var.aws_account_id
@@ -113,6 +115,8 @@ module "psoxy" {
   non_production_connectors            = var.non_production_connectors
   custom_api_connector_rules           = var.custom_api_connector_rules
   lookup_table_builders                = var.lookup_table_builders
+  pseudonymize_app_ids                 = var.pseudonymize_app_ids
+  email_canonicalization               = var.email_canonicalization
   general_environment_variables        = var.general_environment_variables
   function_env_kms_key_arn             = var.project_aws_kms_key_arn
   logs_kms_key_arn                     = var.project_aws_kms_key_arn
@@ -131,11 +135,11 @@ module "psoxy" {
   todo_step                            = local.max_auth_todo_step
 
 
-#  vpc_config = {
-#    vpc_id             = aws_default_vpc.default.id
-#    security_group_ids = [aws_security_group.default.id]
-#    subnet_ids         = [aws_default_subnet.default.id]
-#  }
+  #  vpc_config = {
+  #    vpc_id             = aws_default_vpc.default.id
+  #    security_group_ids = [aws_security_group.default.id]
+  #    subnet_ids         = [aws_default_subnet.default.id]
+  #  }
 }
 
 ## Worklytics connection configuration
@@ -151,7 +155,7 @@ module "connection_in_worklytics" {
   for_each = local.all_instances
 
   source = "../../modules/worklytics-psoxy-connection-aws"
-  # source = "git::https://github.com/worklytics/psoxy//infra/modules/worklytics-psoxy-connection-aws?ref=v0.4.51"
+  # source = "git::https://github.com/worklytics/psoxy//infra/modules/worklytics-psoxy-connection-aws?ref=v0.4.52"
 
   psoxy_instance_id  = each.key
   worklytics_host    = var.worklytics_host

infra/examples-dev/aws-all/msft-365.tf

@@ -2,7 +2,7 @@
 
 module "worklytics_connectors_msft_365" {
   source = "../../modules/worklytics-connectors-msft-365"
-  # source = "git::https://github.com/worklytics/psoxy//infra/modules/worklytics-connectors-msft-365?ref=v0.4.51"
+  # source = "git::https://github.com/worklytics/psoxy//infra/modules/worklytics-connectors-msft-365?ref=v0.4.52"
 
 
   enabled_connectors                  = var.enabled_connectors
@@ -47,7 +47,7 @@ module "cognito_identity_pool" {
   count = local.msft_365_enabled ? 1 : 0 # only provision identity pool if MSFT-365 connectors are enabled
 
   source = "../../modules/aws-cognito-pool"
-  # source = "git::https://github.com/worklytics/psoxy//infra/modules/aws-cognito-pool?ref=v0.4.51"
+  # source = "git::https://github.com/worklytics/psoxy//infra/modules/aws-cognito-pool?ref=v0.4.52"
 
   developer_provider_name = local.developer_provider_name
   name                    = "${local.env_qualifier}-azure-ad-federation"
@@ -57,7 +57,7 @@ module "cognito_identity" {
   count = local.msft_365_enabled ? 1 : 0 # only provision identity pool if MSFT-365 connectors are enabled
 
   source = "../../modules/aws-cognito-identity-cli"
-  # source = "git::https://github.com/worklytics/psoxy//infra/modules/aws-cognito-identity-cli?ref=v0.4.51"
+  # source = "git::https://github.com/worklytics/psoxy//infra/modules/aws-cognito-identity-cli?ref=v0.4.52"
 
   aws_region       = data.aws_region.current.id
   aws_role         = var.aws_assume_role_arn
@@ -79,7 +79,7 @@ module "msft_connection_auth_federation" {
   for_each = module.worklytics_connectors_msft_365.enabled_api_connectors
 
   source = "../../modules/azuread-federated-credentials"
-  # source = "git::https://github.com/worklytics/psoxy//infra/modules/azuread-federated-credentials?ref=v0.4.51"
+  # source = "git::https://github.com/worklytics/psoxy//infra/modules/azuread-federated-credentials?ref=v0.4.52"
 
   application_object_id = each.value.connector.id
   display_name          = "${local.env_qualifier}AccessFromAWS"

infra/examples-dev/aws-all/variables.tf

@@ -181,6 +181,12 @@ variable "pseudonymize_app_ids" {
   default     = true
 }
 
+variable "email_canonicalization" {
+  type        = string
+  description = "defines how email address are processed prior to hashing, hence which are considered 'canonically equivalent'; one of 'STRICT' (default and most standard compliant) or 'IGNORE_DOTS' (probably most in line with user expectations)"
+  default     = "IGNORE_DOTS"
+}
+
 variable "enabled_connectors" {
   type        = list(string)
   description = "list of ids of connectors to enabled; see modules/worklytics-connector-specs"

infra/examples-dev/aws-google-workspace/main.tf

@@ -59,7 +59,7 @@ data "google_project" "psoxy-google-connectors" {
 
 module "psoxy" {
   source = "../../modular-examples/aws-google-workspace"
-  # source = "git::https://github.com/worklytics/psoxy//infra/modular-examples/aws-google-workspace?ref=v0.4.51"
+  # source = "git::https://github.com/worklytics/psoxy//infra/modular-examples/aws-google-workspace?ref=v0.4.52"
 
   aws_account_id                 = var.aws_account_id
   aws_assume_role_arn            = var.aws_assume_role_arn # role that can test the instances (lambdas)

infra/examples-dev/aws-msft-365/main.tf

@@ -51,7 +51,7 @@ provider "azuread" {
 
 module "psoxy" {
   source = "../../modular-examples/aws-msft-365"
-  # source = "git::https://github.com/worklytics/psoxy//infra/modular-examples/aws-msft-365?ref=v0.4.51"
+  # source = "git::https://github.com/worklytics/psoxy//infra/modular-examples/aws-msft-365?ref=v0.4.52"
 
   aws_account_id                 = var.aws_account_id
   aws_assume_role_arn            = var.aws_assume_role_arn # role that can test the instances (lambdas)

infra/examples-dev/aws/main.tf

@@ -57,7 +57,7 @@ provider "azuread" {
 
 module "psoxy" {
   source = "../../modular-examples/aws"
-  # source = "git::https://github.com/worklytics/psoxy//infra/modular-examples/aws?ref=v0.4.51"
+  # source = "git::https://github.com/worklytics/psoxy//infra/modular-examples/aws?ref=v0.4.52"
 
   aws_account_id                 = var.aws_account_id
   aws_assume_role_arn            = var.aws_assume_role_arn # role that can test the instances (lambdas)

infra/examples-dev/gcp-google-workspace/main.tf

@@ -28,7 +28,7 @@ provider "google" {
 
 module "psoxy" {
   source = "../../modular-examples/gcp-google-workspace"
-  # source = "git::https://github.com/worklytics/psoxy//infra/modular-examples/gcp-google-workspace?ref=v0.4.51"
+  # source = "git::https://github.com/worklytics/psoxy//infra/modular-examples/gcp-google-workspace?ref=v0.4.52"
 
   gcp_project_id                 = var.gcp_project_id
   environment_name               = var.environment_name

infra/examples-dev/gcp/google-workspace.tf

@@ -8,7 +8,7 @@ provider "google" {
 
 module "worklytics_connectors_google_workspace" {
   source = "../../modules/worklytics-connectors-google-workspace"
-  # source = "git::https://github.com/worklytics/psoxy//infra/modules/worklytics-connectors-google-workspace?ref=v0.4.51"
+  # source = "git::https://github.com/worklytics/psoxy//infra/modules/worklytics-connectors-google-workspace?ref=v0.4.52"
 
   providers = {
     google = google.google_workspace