[Fix] remove pause_status for lakehouse monitoring to fix updatemonitor error

[xinyijiang-db]

Changes

remove the pause_status from the terraform

Tests

• make test run locally
• relevant change in docs/ folder
• covered with integration tests in internal/acceptance
• relevant acceptance tests are passing
• using Go SDK

[alexott]

I think that we need to go standard deprecation route in this case - if the field is removed from the corresponding API struct, then we need to add it to schema manually and mark it as deprecated, and remove in 2-3 releases after that.

[xinyijiang-db]

I think that we need to go standard deprecation route in this case - if the field is removed from the corresponding API struct, then we need to add it to schema manually and mark it as deprecated, and remove in 2-3 releases after that.

hi @alexott this is a fix because the pause_status is not supposed to be in the schema.
when we update, we do a read before we do a read then update, and having the pause_status causes the update function to fail because our service don't allow pause_status to be passed in the request

also cc @arpitjasa-db who was in the initial discussion with the LHM team and suggested the fix

[alexott]

@xinyijiang-db pause status is still in the API spec: https://docs.databricks.com/api/workspace/qualitymonitors/create#schedule-pause_status, so it's automatically pulled into Go SDK and used. Also, Update REST API has it: https://docs.databricks.com/api/workspace/qualitymonitors/update#schedule-pause_status

I agree with @tanmay-db that it doesn't remove this field from the schema... If you need to remove it completely - make sure you deprecate it in the API spec

[xinyijiang-db]

@alexott let me clarify a little bit more. we still want it in the response so that users can see it but we don't want it in the update function request to our service

it seems that terraform does a read before it calls our update function, which causes our service to reject the update request

in terms of breaking change, this update request to LHM haven't succeeded for any existing users because our service doesn't allow request with the pause_status field coming in, which is being filled in Terraform

[arpitjasa-db]

@xinyijiang-db pause status is still in the API spec: https://docs.databricks.com/api/workspace/qualitymonitors/create#schedule-pause_status, so it's automatically pulled into Go SDK and used. Also, Update REST API has it: https://docs.databricks.com/api/workspace/qualitymonitors/update#schedule-pause_status

I agree with @tanmay-db that it doesn't remove this field from the schema... If you need to remove it completely - make sure you deprecate it in the API spec

I think the idea is that this field doesn't need to be managed by Terraform, and LHM team has had some folks report issues because this field is being sent back in the Update API, so it's easier to just have Terraform ignore this field. However, this field is already marked as ReadOnly in the code so not sure why is this happening? Is this an issue with ReadOnly then?

In terms of a breaking change, folks shouldn't be able to set this field today in their HCL config right? Curious if it does break existing users then?

[alexott]

The problem is that Create and Update APIs have this field as part of the payload, so TF just sends it. I suggest to fix the API spec itself, not the terraform

[alexott]

We need to add or modify integration tests with the schedule block defined, so we can make sure that change actually works

[github-actions]

If integration tests don't run automatically, an authorized user can run them manually by following the instructions below:

Trigger:
go/deco-tests-run/terraform

Inputs:

• PR number: 4224
• Commit SHA: 7356dd17553250ab5a0dee3d7d4fc8eae8b481b6

Checks will be approved automatically on success.

[eng-dev-ecosystem-bot]

Test Details: go/deco-tests/11898706176