Merge pull request #116 from databricks/ai-branch

aws gov simplicity update

aws-gov/customizations/account/logging_configuration/logging_configuration.md

@@ -0,0 +1,22 @@
+### Audit and Billable Usage Logs: 
+Databricks delivers logs to your S3 buckets. [Audit logs](https://docs.databricks.com/administration-guide/account-settings/audit-logs.html) contain two levels of events: workspace-level audit logs with workspace-level events, and account-level audit logs with account-level events. Additionally, you can generate more detailed events by enabling verbose audit logs. [Billable usage logs](https://docs.databricks.com/administration-guide/account-settings/billable-usage-delivery.html) are delivered daily to an AWS S3 storage bucket. A separate CSV file is created for each workspace, containing historical data about the workspace’s cluster usage in Databricks Units (DBUs).
+
+### How to add this resource to SRA:
+
+1. Copy the `logging_configuration` folder into `modules/sra/databricks_account/` 
+2. Add the following code block into `modules/sra/databricks_account.tf`
+```
+module "log_delivery" {
+  source = "./databricks_account/logging_configuration"
+  providers = {
+    databricks = databricks.mws
+  }
+
+  databricks_account_id = var.databricks_account_id
+  resource_prefix       = var.resource_prefix
+}
+```
+3. Run `terraform init`
+4. Run `terraform validate`
+5. From `tf` directory, run `terraform plan -var-file ../example.tfvars`
+6. Run `terraform apply -var-file ../example.tfvars`

aws-gov/customizations/customizations.md

@@ -0,0 +1,21 @@
+# Customizations
+
+Customizations are **Terraform code** available to support the baseline deployment of the **Security Reference Architecture (SRA) - Terraform Templates**.
+
+Customizations are sectioned by providers:
+- **Account**: Databricks account provider.
+- **Workspace**: Databricks workspace provider.
+- **Networking Configuration**: AWS provider.
+
+The current customizations available are:
+
+| Provider                    | Customization                 | Summary |
+|-----------------------------|-------------------------------|---------|
+| **Account** | **Audit and Billable Usage Logs** | Databricks delivers logs to your S3 buckets. [Audit logs](https://docs.databricks.com/administration-guide/account-settings/audit-logs.html) contain workspace and account-level events. Additionally, [Billable usage logs](https://docs.databricks.com/administration-guide/account-settings/billable-usage-delivery.html) are delivered daily to an AWS S3 bucket, containing historical data about cluster usage in DBUs for each workspace. |
+| **Workspace** | **Workspace Admin Configurations** | Workspace administration configurations can be enabled to align with security best practices. The Terraform resource is experimental and optional, with documentation on each configuration provided in the Terraform file. |
+| **Workspace** | **IP Access Lists** | IP Access can be enabled to restrict console access to a subset of IPs. **NOTE:** Ensure IPs are accurate to prevent lockout scenarios. |
+| **Workspace** | **Security Analysis Tool (SAT)** | The Security Analysis Tool evaluates a customer’s Databricks account and workspace security configurations, providing recommendations that align with Databricks’ best practices. This can be enabled within the workspace. |
+| **Workspace** | **Audit Log Alerting** | Based on this [blog post](https://www.databricks.com/blog/improve-lakehouse-security-monitoring-using-system-tables-databricks-unity-catalog), Audit Log Alerting creates 40+ SQL alerts to monitor incidents following a Zero Trust Architecture (ZTA) model. **NOTE:** This configuration creates a cluster, a job, and queries within your environment. |
+| **Workspace** | **Read-Only External Location** | Creates a read-only external location in Unity Catalog for a specified bucket, as well as the corresponding AWS IAM role. |
+| **AWS** | **Firewall (Limited Egress)** | This firewall networking configuration restricts traffic to a specified list of public addresses, suitable for situations where open internet access is limited due to workload or data sensitivity. |
+| **AWS** | **Sandbox (Open Egress)** | Open egress allows free traffic flow to the public internet, ideal for sandbox or development scenarios where data exfiltration protection is minimal, and developers need public API access. |

aws-gov/customizations/networking_configurations/firewall/firewall.md

@@ -0,0 +1,66 @@
+### Firewall (Limited Egress)
+Using a firewall networking configuration restricts traffic flow to a specified list of public addresses. This setup is applicable in situations where open internet access is necessary for certain tasks, but unfiltered traffic is not an option due to workload or data sensitivity.
+
+- **WARNING**: Due to a limitation in AWS Network Firewall's support for fully qualified domain names (FQDNs) in non-HTTP/HTTPS traffic, an IP address is required to allow communication with the Hive Metastore. This reliance on a static IP introduces the potential for downtime if the Hive Metastore's IP changes. For sensitive production workloads, it is recommended to explore the isolated operation mode or consider alternative firewall solutions that provide better handling of dynamic IPs or FQDNs.
+
+Please note that a public and firewall subnet CIDR ranges are required.
+
+### How to use this network configuration:
+1. Replace the VPC module (lines 4-28) in `modules/sra/network.tf` with the following code block:
+```
+module "vpc" {
+  source  = "terraform-aws-modules/vpc/aws"
+  version = "5.1.1"
+
+  count = 1
+
+  name = "${var.resource_prefix}-classic-compute-plane-vpc"
+  cidr = var.vpc_cidr_range
+  azs  = var.availability_zones
+
+  enable_dns_hostnames   = true
+  enable_nat_gateway     = false
+  single_nat_gateway     = false
+  one_nat_gateway_per_az = false
+  create_igw             = false
+
+  public_subnet_names = []
+  public_subnets      = []
+
+  private_subnet_names = [for az in var.availability_zones : format("%s-private-%s", var.resource_prefix, az)]
+  private_subnets      = var.private_subnets_cidr
+
+  intra_subnet_names = [for az in var.availability_zones : format("%s-privatelink-%s", var.resource_prefix, az)]
+  intra_subnets      = var.privatelink_subnets_cidr
+
+  tags = {
+    Project = var.resource_prefix
+  }
+}
+```
+2. Create a new directory `modules/sra/firewall`
+3. Copy the `firewall` directory from `networking_configurations/firewall` into the new directory.
+3. Create a new file `modules/sra/firewall.tf`
+4. Add the following code block into `modules/sra/firewall.tf`
+```
+module "harden_firewall" {
+  source = "./firewall"
+  providers = {
+    aws = aws
+  }
+
+  vpc_id                = module.vpc[0].vpc_id
+  vpc_cidr_range        = var.vpc_cidr_range
+  public_subnets_cidr   = <list(string) of public subnet CIDR ranges>
+  private_subnets_cidr  = module.vpc[0].private_subnets_cidr_blocks
+  private_subnet_rt     = module.vpc[0].private_route_table_ids
+  firewall_subnets_cidr = <list(string) of firewall subnet CIDR ranges>
+  firewall_allow_list   = <list(string) of allowed FQDNs>
+  hive_metastore_fqdn   = var.hive_metastore_fqdn[var.databricks_gov_shard] // <HMS FQDN from: https://docs.databricks.com/en/resources/ip-domain-region.html#rds-addresses-for-legacy-hive-metastore>
+  availability_zones    = var.availability_zones
+  region                = var.region
+  resource_prefix       = var.resource_prefix
+
+  depends_on = [module.databricks_mws_workspace]
+}
+```

aws-gov/tf/modules/sra/data_plane_hardening/firewall/firewall.tf → aws-gov/customizations/networking_configurations/firewall/firewall.tf

@@ -187,7 +187,7 @@ resource "aws_networkfirewall_rule_group" "databricks_fqdn_allowlist" {
 }
 
 data "dns_a_record_set" "metastore_dns" {
-  host     = var.hive_metastore_fqdn
+  host = var.hive_metastore_fqdn[var.databricks_gov_shard]
 }
 
 // JDBC Firewall group IP allow list
@@ -205,7 +205,7 @@ resource "aws_networkfirewall_rule_group" "databricks_metastore_allowlist" {
         content {
           action = "PASS"
           header {
-              destination      = stateful_rule.value
+            destination      = stateful_rule.value
             destination_port = 3306
             direction        = "FORWARD"
             protocol         = "TCP"

aws-gov/tf/modules/sra/data_plane_hardening/firewall/variables.tf → aws-gov/customizations/networking_configurations/firewall/variables.tf

@@ -11,7 +11,11 @@ variable "firewall_subnets_cidr" {
 }
 
 variable "hive_metastore_fqdn" {
-  type = string
+  type = map(string)
+  default = {
+    "civilian" = "discovery-search-rds-prod-dbdiscoverysearch-uus7j2cyyu1m.c40ji7ukhesx.us-gov-west-1.rds.amazonaws.com"
+    "dod"      = "lineage-usgovwest1dod-prod.cpnejponioft.us-gov-west-1.rds.amazonaws.com"
+  }
 }
 
 variable "private_subnet_rt" {
@@ -40,4 +44,8 @@ variable "vpc_cidr_range" {
 
 variable "vpc_id" {
   type = string
+}
+
+variable "databricks_gov_shard" {
+  type = string
 }

aws-gov/customizations/networking_configurations/sandbox/sandbox.md

@@ -0,0 +1,41 @@
+### Sandbox (Open Egress)
+Using open egress allows traffic to flow freely to the public internet. This networking configuration is suitable for sandbox or development scenarios where data exfiltration protection is of minimal concern, and developers need access to public APIs, packages, and more.
+
+Please note that a public subnet CIDR range is required.
+
+
+### How to use this network configuration:
+
+1. Replace the VPC module (lines 4-28) in `modules/sra/network.tf` with the following code block:
+```
+module "vpc" {
+  source  = "terraform-aws-modules/vpc/aws"
+  version = "5.1.1"
+
+  count = 1
+
+  name = "${var.resource_prefix}-classic-compute-plane-vpc"
+  cidr = var.vpc_cidr_range
+  azs  = var.availability_zones
+
+  enable_dns_hostnames   = true
+  enable_nat_gateway     = true
+  single_nat_gateway     = false
+  one_nat_gateway_per_az = true
+  create_igw             = true
+
+  public_subnet_names = [for az in var.availability_zones : format("%s-public-%s", var.resource_prefix, az)]
+  public_subnets      = <list(string) of public subnet CIDR ranges>
+
+  private_subnet_names = [for az in var.availability_zones : format("%s-private-%s", var.resource_prefix, az)]
+  private_subnets      = var.private_subnets_cidr
+
+  intra_subnet_names = [for az in var.availability_zones : format("%s-privatelink-%s", var.resource_prefix, az)]
+  intra_subnets      = var.privatelink_subnets_cidr
+
+  tags = {
+    Project = var.resource_prefix
+  }
+}
+
+```

aws-gov/customizations/workspace/admin_configuration/admin_configuration.md

@@ -0,0 +1,19 @@
+### **Workspace Admin. Configurations**: 
+Workspace administration configurations can be enabled to align with security best practices. This Terraform resource is experimental, making it optional. Documentation for each configuration is provided in the Terraform file.
+
+### How to add this resource to SRA:
+
+1. Copy the `admin_configuration` folder into `modules/sra/databricks_workspace/` 
+2. Add the following code block into `modules/sra/databricks_workspace.tf`
+```
+module "admin_configuration" {
+  source = "./databricks_workspace/admin_configuration"
+  providers = {
+    databricks = databricks.created_workspace
+  }
+}
+```
+3. Run `terraform init`
+4. Run `terraform validate`
+5. From `tf` directory, run `terraform plan -var-file ../example.tfvars`
+6. Run `terraform apply -var-file ../example.tfvars`

aws-gov/customizations/workspace/ip_access_list/ip_access_list.md

@@ -0,0 +1,23 @@
+### IP Access Lists
+IP Access can be enabled to restrict access to the Databricks workspace console to a specified subset of IPs.
+- **NOTE:** Verify that all IPs are correct before enabling this feature to prevent a lockout scenario.
+
+
+### How to add this resource to SRA:
+
+1. Copy the `ip_access_list` folder into `modules/sra/databricks_workspace/` 
+2. Add the following code block into `modules/sra/databricks_workspace.tf`
+```
+module "ip_access_list" {
+    source = "./databricks_workspace/ip_access_list"
+    providers = {
+    databricks = databricks.created_workspace
+  }
+
+    ip_addresses = <list(string) of IP addresses>
+}
+```
+3. Run `terraform init`
+4. Run `terraform validate`
+5. From `tf` directory, run `terraform plan -var-file ../example.tfvars`
+6. Run `terraform apply -var-file ../example.tfvars`

aws-gov/customizations/workspace/solution_accelerators/security_analysis_tool/aws/TERRAFORM_AWS.md

@@ -0,0 +1,87 @@
+## Setting up Terraform
+
+> **SAT v0.2.0 or higher** brings full support for Unity Catalog. Now you can pick your catalog instead of hive_metastore. Plus, you get to choose your own schema name.
+
+Step 1: [Install Terraform](https://developer.hashicorp.com/terraform/tutorials/aws-get-started/install-cli)
+
+Step 2: [Install Git](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git) on local machine
+
+Step 3: Git Clone Repo
+
+  ```sh
+    git clone https://github.com/databricks-industry-solutions/security-analysis-tool.git
+   ```
+
+Step 4: Change Directories
+
+  ```sh
+    cd security-analysis-tool/terraform/<cloud>/
+  ```
+
+Step 5: Generate a `terraform.tfvars` file base on `template.tfvars`
+
+Using any editor set the values in the `terraform.tfvars` file. The descriptions of all the variables are located in the variables.tf file. Once the variables are set you are ready to run Terraform.
+
+Further Documentation for some of the variables:
+
+[workspace_id](https://docs.databricks.com/workspace/workspace-details.html#workspace-instance-names-urls-and-ids)
+
+[account_console_id](https://docs.databricks.com/administration-guide/account-settings/index.html#locate-your-account-id)
+
+> **Proxies are now supported as part of SAT. You can add your HTTP and HTTPS links to use your proxies.**
+```json
+{
+    "http": "http://example.com",
+    "https": "https://example.com"
+}
+```
+
+## Run Terraform
+
+Step 6: Terraform [Init](https://developer.hashicorp.com/terraform/cli/commands/init)
+
+The terraform init command initializes a working directory containing configuration files and installs plugins for required providers.
+
+  ```sh
+  terraform init
+  ```
+
+Step 7: Terraform [Plan](https://developer.hashicorp.com/terraform/cli/commands/plan)
+
+The terraform plan command creates an execution plan, which lets you preview the changes that Terraform plans to make to your infrastructure. By default, when Terraform creates a plan it:
+
+* Reads the current state of any already-existing remote objects to make sure that the Terraform state is up-to-date.
+* Compares the current configuration to the prior state and noting any differences.
+* Proposes a set of change actions that should, if applied, make the remote objects match the configuration.
+
+  ```sh
+  terraform plan
+  ```
+
+Step 8: Terraform [Apply](https://developer.hashicorp.com/terraform/cli/commands/apply)
+
+The terraform apply command executes the actions proposed in a Terraform plan.
+
+  ```
+    terraform apply
+  ```
+
+Step 9: Run Jobs
+
+You now have two jobs (SAT Initializer Notebook & SAT Driver Notebook). Run SAT Initializer Notebook and when it completes run SAT Driver Notebook; SAT Initializer Notebook should only be run once (although you can run it multiple times, it only needs to be run successfully one time), and SAT Driver Notebook can be run periodically (its scheduled to run once every Monday, Wednesday, and Friday).
+
+Step 10: SAT Dashboard
+
+Go to the SQL persona, select the Dashboard icon in the left menu and then select the SAT Dashboard. Once the dashboard loads pick the Workspace from the dropdown and refresh the dashboard
+
+Supplemental Documentation:
+
+[Databricks Documentation Terraform](https://docs.databricks.com/dev-tools/terraform/index.html)
+
+[Databricks Terraform Provider Docs](https://registry.terraform.io/providers/databricks/databricks/latest/docs)
+
+Additional Considerations:
+
+Your jobs may fail if there was a pre-existing secret scope named sat_scope when you run terraform apply. To remedy this, you will need to change the name of your secret scope in secrets.tf, re-run terraform apply, and then navigate to Workspace -> Applications -> SAT-TF ->/notebooks/Utils/initialize and change the secret scope name in  6 places (3 times in CMD 4 and 3 times in CMD 5). You then can re-run your failed jobs.
+
+Congratulations!!! [Please review the setup documentation for the instructions on usage, FAQs and general understanding of SAT setup](https://github.com/databricks-industry-solutions/security-analysis-tool/blob/main/docs/setup.md)

aws-gov/customizations/workspace/solution_accelerators/security_analysis_tool/common/jobs.tf

@@ -0,0 +1,40 @@
+resource "databricks_job" "initializer" {
+  name = "SAT Initializer Notebook (one-time)"
+  environment {
+    spec {
+      dependencies = ["dbl-sat-sdk"]
+      client       = "1"
+    }
+    environment_key = "Default"
+  }
+  task {
+    task_key        = "Initializer"
+    notebook_task {
+      notebook_path = "${databricks_repo.security_analysis_tool.workspace_path}/notebooks/security_analysis_initializer"
+    }
+  }
+}
+
+resource "databricks_job" "driver" {
+  name = "SAT Driver Notebook"
+  environment {
+    spec {
+      dependencies = ["dbl-sat-sdk"]
+      client       = "1"
+    }
+    environment_key = "Default"
+  }
+  task {
+    task_key        = "Driver"
+    notebook_task {
+      notebook_path = "${databricks_repo.security_analysis_tool.workspace_path}/notebooks/security_analysis_driver"
+    }
+  }
+
+  schedule {
+    #E.G. At 08:00:00am, on every Monday, Wednesday and Friday, every month; For more: http://www.quartz-scheduler.org/documentation/quartz-2.3.0/tutorials/crontrigger.html
+    quartz_cron_expression = "0 0 8 ? * Mon,Wed,Fri"
+    # The system default is UTC; For more: https://en.wikipedia.org/wiki/List_of_tz_database_time_zones
+    timezone_id = "America/New_York"
+  }
+}

aws-gov/tf/modules/sra/databricks_workspace/solution_accelerators/security_analysis_tool/common/variables.tf → aws-gov/customizations/workspace/solution_accelerators/security_analysis_tool/common/variables.tf

@@ -29,12 +29,6 @@ variable "notification_email" {
   default     = ""
 }
 
-variable "gcp_impersonate_service_account" {
-  type        = string
-  description = "GCP Service Account to impersonate (e.g. [email protected])"
-  default     = ""
-}
-
 variable "analysis_schema_name" {
   type        = string
   description = "Name of the schema to be used for analysis"