Skip to content

Commit

Permalink
merge
Browse files Browse the repository at this point in the history
  • Loading branch information
jdavidheiser committed Feb 22, 2024
2 parents cd72462 + 9ec61f2 commit 0c73b51
Show file tree
Hide file tree
Showing 190 changed files with 6,281 additions and 1,467 deletions.
6 changes: 4 additions & 2 deletions .github/ISSUE_TEMPLATE/docs-issue.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,10 +16,12 @@ This template is for both adding enhancement as well as pointing out issues with
### Expected Details
<!-- What are you expecting from the section that has an issue. If the section is missing anything that should be expected please point that out. -->

### List of things to potentially add/remove:
### List of things to potentially add/remove

This is a list of things to manipulate in the docs:

- [ ] First item to change
- [ ] Second item to change
- [ ] Second item to change

### Important Factoids
<!-- Any links to external documentation that may prove your case, i.e Databricks public docs or Terraform public docs. -->
Expand Down
1 change: 0 additions & 1 deletion .github/PULL_REQUEST_TEMPLATE.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,4 +11,3 @@ How is this tested? Please see the checklist below and also describe any other r
- [ ] covered with integration tests in `internal/acceptance`
- [ ] relevant acceptance tests are passing
- [ ] using Go SDK

117 changes: 117 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,122 @@
# Version changelog

## 1.37.1

### New Features and Improvements
* Removed `CustomizeDiff` and Client Side Validation for [databricks_grants](https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/grants) ([#3290](https://github.com/databricks/terraform-provider-databricks/pull/3290)).
* Added terraform support for restrict ws admins setting ([#3243](https://github.com/databricks/terraform-provider-databricks/pull/3243)).

### Internal Changes
* Migrated [databricks_global_init_script](https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/global_init_script) to Go SDK ([#2036](https://github.com/databricks/terraform-provider-databricks/pull/2036)).
* Bump github.com/hashicorp/terraform-plugin-sdk/v2 from 2.31.0 to 2.32.0 ([#3177](https://github.com/databricks/terraform-provider-databricks/pull/3177)).


## 1.37.0

### New Features and Improvements
* Add [databricks_file](https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/file) resource ([#3265](https://github.com/databricks/terraform-provider-databricks/pull/3265)).
* Add [databricks_storage_credential](https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/storage_credential) and [databricks_storage_credentials](https://registry.terraform.io/providers/databricks/databricks/latest/docs/data-sources/storage_credentials) data sources ([#3254](https://github.com/databricks/terraform-provider-databricks/pull/3254)).
* Add `source` attribute to `dbt_task` and `sql_task.file` tasks to support files from workspace ([#3208](https://github.com/databricks/terraform-provider-databricks/pull/3208)).
* Add computed `volume_path` attribute to [databricks_volume](https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/volume) resource ([#3272](https://github.com/databricks/terraform-provider-databricks/pull/3272)).
* Add support for Vector Search Endpoints ([#3191](https://github.com/databricks/terraform-provider-databricks/pull/3191)).
* [JOBS-16324] Terraform support for Foreach tasks (private preview) ([#3252](https://github.com/databricks/terraform-provider-databricks/pull/3252)).
* fix: properly propagate auth_type to the databricks client ([#3273](https://github.com/databricks/terraform-provider-databricks/pull/3273)).

### Documentation Changes
* Fix images and add note on setting multiple authorizations for workspace setup ([#3259](https://github.com/databricks/terraform-provider-databricks/pull/3259)).
* Remove `data_object_type=TABLE` only restriction in [databricks_share](https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/share) ([#3108](https://github.com/databricks/terraform-provider-databricks/pull/3108)).
* Remove legacy guides ([#3282](https://github.com/databricks/terraform-provider-databricks/pull/3282)).
* Update `for_each_task` docs. ([#3271](https://github.com/databricks/terraform-provider-databricks/pull/3271)).

### Exporter
* Support for some Unity Catalog resources ([#3242](https://github.com/databricks/terraform-provider-databricks/pull/3242)).
* Rework handling of listings and interactive prompting ([#3241](https://github.com/databricks/terraform-provider-databricks/pull/3241)).
* UC exporter databricks storage credential feature ([#3219](https://github.com/databricks/terraform-provider-databricks/pull/3219)).

### Internal Changes
* Add CustomDiffFunc for health in sql_endpoint resources ([#3227](https://github.com/databricks/terraform-provider-databricks/pull/3227)).
* Bump github.com/databricks/databricks-sdk-go 0.33.0 ([#3275](https://github.com/databricks/terraform-provider-databricks/pull/3275)).
* Suppress diff on whitespace change for resources that often use HERE-docs ([#3251](https://github.com/databricks/terraform-provider-databricks/pull/3251)).


## 1.36.3

### New Features and Improvements
* Explicitly set securable field when reading [databricks_grants](https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/grants) or [databricks_grant](https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/grant) ([#3246](https://github.com/databricks/terraform-provider-databricks/pull/3246)).

### Documentation Changes
* Added information on `id` and other exposed attributes where appropriate ([#3237](https://github.com/databricks/terraform-provider-databricks/pull/3237)).
* Fixed docs in metastore `databricks_grants` example ([#3239](https://github.com/databricks/terraform-provider-databricks/pull/3239)).

### Exporter
* Detect & handle deleted workspace objects (notebooks/files/directories) when running in incremental mode ([#3225](https://github.com/databricks/terraform-provider-databricks/pull/3225)).

### Internal Changes
* Make IterFields take in aliases ([#3207](https://github.com/databricks/terraform-provider-databricks/pull/3207)).


## 1.36.2

### New Features and Improvements
* Added [databricks_aws_unity_catalog_policy](https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/aws_unity_catalog_policy) data source ([#2483](https://github.com/databricks/terraform-provider-databricks/pull/2483)).
* Removed `omitempty` in `destination` fields in `clustes_api.go` ([#3232](https://github.com/databricks/terraform-provider-databricks/pull/3232)), to address ([#3231](https://github.com/databricks/terraform-provider-databricks/issues/3231))

### Exporter
* Omitted `git_provider` only for well-known Git URLs ([#3216](https://github.com/databricks/terraform-provider-databricks/pull/3216)).

### Internal Changes
* Bumped github.com/zclconf/go-cty from 1.14.1 to 1.14.2 ([#3144](https://github.com/databricks/terraform-provider-databricks/pull/3144)).
* Bumped golang.org/x/mod from 0.14.0 to 0.15.0 ([#3229](https://github.com/databricks/terraform-provider-databricks/pull/3229)).


## 1.36.1

### New Features and Improvements
* Fixed create storage credentials with owner for account ([#3184](https://github.com/databricks/terraform-provider-databricks/pull/3184)).

### Documentation Changes
* Removed AWS-only note for [databricks_service_principal_secret](https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/service_principal_secret) resource ([#3213](https://github.com/databricks/terraform-provider-databricks/pull/3213)).

### Internal Changes
* Fixed test: TestUcAccResourceSqlTable_Managed ([#3226](https://github.com/databricks/terraform-provider-databricks/pull/3226)).

## 1.36.0

### New Features and Improvements
* Added `databricks_volumes` as data source ([#3150](https://github.com/databricks/terraform-provider-databricks/pull/3150)).
* Fixed updating owners for UC resources ([#3189](https://github.com/databricks/terraform-provider-databricks/pull/3189)).
* Validated metastore id for databricks_grant and databricks_grants resources ([#3159](https://github.com/databricks/terraform-provider-databricks/pull/3159)).
* Fixed `databricks_connection` regression when creating without owner ([#3186](https://github.com/databricks/terraform-provider-databricks/pull/3186)).
* Allow using empty strings as job parameters ([#3158](https://github.com/databricks/terraform-provider-databricks/pull/3158)).
* Changed type of value field of `JobsHealthRule` to `int64` ([#3215](https://github.com/databricks/terraform-provider-databricks/pull/3215)).


### Documentation Changes
* Various documentation updates ([#3198](https://github.com/databricks/terraform-provider-databricks/pull/3198)).
* Fixed typo in docs ([#3166](https://github.com/databricks/terraform-provider-databricks/pull/3166)).

### Exporter
* Timestamps are now added to log entries ([#3146](https://github.com/databricks/terraform-provider-databricks/pull/3146)).
* Add retries for `Search`, `ReadContext` and `Import` operations when importing the resource ([#3202](https://github.com/databricks/terraform-provider-databricks/pull/3202)).
* Performance improvements for big workspaces ([#3167](https://github.com/databricks/terraform-provider-databricks/pull/3167)).
* Fix generation of cluster policy resources ([#3185](https://github.com/databricks/terraform-provider-databricks/pull/3185)).
* Skip emitting of clusters that come from more cluster sources ([#3161](https://github.com/databricks/terraform-provider-databricks/pull/3161)).

### Internal Changes
* Migrated cluster schema to use the go-sdk struct ([#3076](https://github.com/databricks/terraform-provider-databricks/pull/3076)).
* Updated actions/setup-go to v5 ([#3154](https://github.com/databricks/terraform-provider-databricks/pull/3154)).
* Changed default branch from `master` to `main` ([#3174](https://github.com/databricks/terraform-provider-databricks/pull/3174)).
* Added .codegen.json configuration ([#3180](https://github.com/databricks/terraform-provider-databricks/pull/3180)).
* Used common.Resource consistently throughout the provider ([#3193](https://github.com/databricks/terraform-provider-databricks/pull/3193)).
* Fixed unit test ([#3201](https://github.com/databricks/terraform-provider-databricks/pull/3201)).
* Added test code for job task order ([#3183](https://github.com/databricks/terraform-provider-databricks/pull/3183)).
* Added unit test for `customizable_schema.go` ([#3192](https://github.com/databricks/terraform-provider-databricks/pull/3192)).
* Extended customizable schema with `AtLeastOneOf`, `ExactlyOneOf`, `RequiredWith` ([#3182](https://github.com/databricks/terraform-provider-databricks/pull/3182)).
* Fixed notebook parameters in acceptance test ([#3205](https://github.com/databricks/terraform-provider-databricks/pull/3205)).
* Introduced Generic Settings Resource ([#2997](https://github.com/databricks/terraform-provider-databricks/pull/2997)).
* Suppress diff should apply to new fields added in the same chained call to CustomizableSchema ([#3200](https://github.com/databricks/terraform-provider-databricks/pull/3200)).


## 1.35.0

### New Features and Improvements:
Expand Down
4 changes: 2 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -167,7 +167,7 @@ To make Databricks Terraform Provider generally available, we've moved it from [

You should have [`.terraform.lock.hcl`](https://github.com/databrickslabs/terraform-provider-databricks/blob/v0.6.2/scripts/versions-lock.hcl) file in your state directory that is checked into source control. terraform init will give you the following warning.

```
```text
Warning: Additional provider information from registry
The remote registry returned warnings for registry.terraform.io/databrickslabs/databricks:
Expand All @@ -178,6 +178,6 @@ After you replace `databrickslabs/databricks` with `databricks/databricks` in th

If you didn't check-in [`.terraform.lock.hcl`](https://www.terraform.io/language/files/dependency-lock#lock-file-location) to the source code version control, you may you may see `Failed to install provider` error. Please follow the simple steps described in the [troubleshooting guide](docs/guides/troubleshooting.md).

```
```text
Warning: Exporter is experimental and provided as is. It has an evolving interface, which may change or be removed in future versions of the provider.
```
107 changes: 107 additions & 0 deletions aws/data_aws_unity_catalog_policy.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,107 @@
package aws

import (
"context"
"encoding/json"
"fmt"
"regexp"

"github.com/databricks/terraform-provider-databricks/common"
"github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema"
"github.com/hashicorp/terraform-plugin-sdk/v2/helper/validation"
)

func generateReadContext(ctx context.Context, d *schema.ResourceData, m *common.DatabricksClient) error {
bucket := d.Get("bucket_name").(string)
awsAccountId := d.Get("aws_account_id").(string)
roleName := d.Get("role_name").(string)
policy := awsIamPolicy{
Version: "2012-10-17",
Statements: []*awsIamPolicyStatement{
{
Effect: "Allow",
Actions: []string{
"s3:GetObject",
"s3:PutObject",
"s3:DeleteObject",
"s3:ListBucket",
"s3:GetBucketLocation",
},
Resources: []string{
fmt.Sprintf("arn:aws:s3:::%s/*", bucket),
fmt.Sprintf("arn:aws:s3:::%s", bucket),
},
},
{
Effect: "Allow",
Actions: []string{
"sts:AssumeRole",
},
Resources: []string{
fmt.Sprintf("arn:aws:iam::%s:role/%s", awsAccountId, roleName),
},
},
},
}
if kmsKey, ok := d.GetOk("kms_name"); ok {
policy.Statements = append(policy.Statements, &awsIamPolicyStatement{
Effect: "Allow",
Actions: []string{
"kms:Decrypt",
"kms:Encrypt",
"kms:GenerateDataKey*",
},
Resources: []string{
fmt.Sprintf("arn:aws:kms:%s", kmsKey),
},
})
}
policyJSON, err := json.MarshalIndent(policy, "", " ")
if err != nil {
return err
}
d.SetId(fmt.Sprintf("%s-%s-%s", bucket, awsAccountId, roleName))
err = d.Set("json", string(policyJSON))
if err != nil {
return err
}
return nil
}

func validateSchema() map[string]*schema.Schema {
return map[string]*schema.Schema{
"kms_name": {
Type: schema.TypeString,
Optional: true,
ValidateFunc: validation.StringMatch(
regexp.MustCompile(`^[0-9a-zA-Z/_-]+$`),
"must contain only alphanumeric, hyphens, forward slashes, and underscores characters"),
},
"bucket_name": {
Type: schema.TypeString,
Required: true,
ValidateFunc: validation.StringMatch(
regexp.MustCompile(`^[0-9a-zA-Z_-]+$`),
"must contain only alphanumeric, underscore, and hyphen characters"),
},
"role_name": {
Type: schema.TypeString,
Required: true,
},
"aws_account_id": {
Type: schema.TypeString,
Required: true,
},
"json": {
Type: schema.TypeString,
Computed: true,
},
}
}

func DataAwsUnityCatalogPolicy() common.Resource {
return common.Resource{
Read: generateReadContext,
Schema: validateSchema(),
}
}
121 changes: 121 additions & 0 deletions aws/data_aws_unity_catalog_policy_test.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,121 @@
package aws

import (
"encoding/json"
"testing"

"github.com/databricks/terraform-provider-databricks/qa"
"github.com/stretchr/testify/assert"
)

func TestDataAwsUnityCatalogPolicy(t *testing.T) {
d, err := qa.ResourceFixture{
Read: true,
Resource: DataAwsUnityCatalogPolicy(),
NonWritable: true,
ID: ".",
HCL: `
aws_account_id = "123456789098"
bucket_name = "databricks-bucket"
role_name = "databricks-role"
kms_name = "databricks-kms"
`,
}.Apply(t)
assert.NoError(t, err)
j := d.Get("json").(string)
p := `{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:GetObject",
"s3:PutObject",
"s3:DeleteObject",
"s3:ListBucket",
"s3:GetBucketLocation"
],
"Resource": [
"arn:aws:s3:::databricks-bucket/*",
"arn:aws:s3:::databricks-bucket"
]
},
{
"Effect": "Allow",
"Action": [
"sts:AssumeRole"
],
"Resource": [
"arn:aws:iam::123456789098:role/databricks-role"
]
},
{
"Effect": "Allow",
"Action": [
"kms:Decrypt",
"kms:Encrypt",
"kms:GenerateDataKey*"
],
"Resource": [
"arn:aws:kms:databricks-kms"
]
}
]
}`
compareJSON(t, j, p)
}

func TestDataAwsUnityCatalogPolicyWithoutKMS(t *testing.T) {
d, err := qa.ResourceFixture{
Read: true,
Resource: DataAwsUnityCatalogPolicy(),
NonWritable: true,
ID: ".",
HCL: `
aws_account_id = "123456789098"
bucket_name = "databricks-bucket"
role_name = "databricks-role"
`,
}.Apply(t)
assert.NoError(t, err)
j := d.Get("json").(string)
p := `{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:GetObject",
"s3:PutObject",
"s3:DeleteObject",
"s3:ListBucket",
"s3:GetBucketLocation"
],
"Resource": [
"arn:aws:s3:::databricks-bucket/*",
"arn:aws:s3:::databricks-bucket"
]
},
{
"Effect": "Allow",
"Action": [
"sts:AssumeRole"
],
"Resource": [
"arn:aws:iam::123456789098:role/databricks-role"
]
}
]
}`
compareJSON(t, j, p)
}

func compareJSON(t *testing.T, json1 string, json2 string) {
var i1 interface{}
var i2 interface{}
err := json.Unmarshal([]byte(json1), &i1)
assert.NoError(t, err, "error while unmarshalling")
err = json.Unmarshal([]byte(json2), &i2)
assert.NoError(t, err, "error while unmarshalling")
assert.Equal(t, i1, i2)
}
Loading

0 comments on commit 0c73b51

Please sign in to comment.