Merge branch 'main' into b/fix-retry-messages-in-cloud-backend-json

CHANGELOG.md

@@ -1,35 +1,4 @@
-## 1.10.0 (Unreleased)
-
-NEW FEATURES:
- - **Ephemeral resources**: Ephemeral resources are read anew during each phase of Terraform evaluation, and cannot be persisted to state storage. Ephemeral resources always produce ephemeral values.
- - **Ephemeral values**: Input variables and outputs can now be defined as ephemeral. Ephemeral values may only be used in certain contexts in Terraform configuration, and are not persisted to the plan or state files.
-    - `terraform output -json` now displays ephemeral outputs. The value of an ephemeral output is always `null` unless a plan or apply is being run. Note that `terraform output` (without the `-json`) flag does not yet display ephemeral outputs.
-    - **`ephemeralasnull` function**: a function takes a value of any type and returns a similar value of the same type with any ephemeral values replaced with non-ephemeral null values and all non-ephemeral values preserved.
-
-BUG FIXES:
-
-- The `secret_suffix` in the `kubernetes` backend now includes validation to prevent errors when the `secret_suffix` ends with a number ([#35666](https://github.com/hashicorp/terraform/pull/35666)).
-- The error message for an invalid default value for an input variable now indicates when the problem is with a nested value in a complex data type. ([#35465](https://github.com/hashicorp/terraform/issues/35465))
-- Sensitive marks could be incorrectly transferred to nested resource values, causing erroneous changes during a plan ([#35501](https://github.com/hashicorp/terraform/issues/35501))
-- Allow unknown `error_message` values to pass the core validate step, so variable validation can be completed later during plan
-  ([#35537](https://github.com/hashicorp/terraform/issues/35537))
-- Unencoded slashes within GitHub module source refs were being truncated and incorrectly used as subdirectories in the request path ([#35552](https://github.com/hashicorp/terraform/issues/35552))
-- Terraform refresh-only plans with output only changes are now applyable. ([#35812](https://github.com/hashicorp/terraform/issues/35812))
-- Postconditions referencing `self` with many instances could encounter an error during evaluation [GH-35895]
-- The `plantimestamp()` function would return an invalid date during validation [GH-35902]
-
-ENHANCEMENTS:
-
-- The `element` function now accepts negative indices ([#35501](https://github.com/hashicorp/terraform/issues/35501))
-- Import block validation has been improved to provide more useful errors and catch more invalid cases during `terraform validate` ([#35543](https://github.com/hashicorp/terraform/issues/35543))
-- Performance enhancements for resource evaluation, especially when large numbers of resource instances are involved ([#35558](https://github.com/hashicorp/terraform/issues/35558))
-- The `plan`, `apply`, and `refresh` commands now produce a deprecated warning when using the `-state` flag. Instead use the `path` attribute within the `local` backend to modify the state file. ([#35660](https://github.com/hashicorp/terraform/issues/35660))
-
-UPGRADE NOTES:
-
-- backend/s3: Removes deprecated attributes for assuming IAM role. Must use the `assume_role` block ([#35721](https://github.com/hashicorp/terraform/issues/35721))
-- backend/s3: The s3 backend now supports S3 native state locking. When used with DynamoDB-based locking, locks will be acquired from both sources. In a future minor release of Terraform the DynamoDB locking mechanism and associated arguments will be deprecated. ([#35661](https://github.com/hashicorp/terraform/issues/35661))
-- `moved`: Moved blocks now respect reserved keywords when parsing resource addresses. Configurations that reference resources with type names that match top level blocks and keywords from `moved` blocks will need to prepend the `resource.` identifier to these references. ([#35850](https://github.com/hashicorp/terraform/issues/35850))
+## 1.11.0 (Unreleased)
 
 EXPERIMENTS:
 
@@ -43,6 +12,7 @@ Experiments are only enabled in alpha releases of Terraform CLI. The following f
 
 For information on prior major and minor releases, refer to their changelogs:
 
+- [v1.10](https://github.com/hashicorp/terraform/blob/v1.10/CHANGELOG.md)
 - [v1.9](https://github.com/hashicorp/terraform/blob/v1.9/CHANGELOG.md)
 - [v1.8](https://github.com/hashicorp/terraform/blob/v1.8/CHANGELOG.md)
 - [v1.7](https://github.com/hashicorp/terraform/blob/v1.7/CHANGELOG.md)

internal/command/views/hook_ui_test.go

@@ -619,7 +619,7 @@ func TestUiHookEphemeralOp_progress(t *testing.T) {
 		t.Fatalf("Expected hook to continue, given: %#v", action)
 	}
 
-	time.Sleep(3100 * time.Millisecond)
+	time.Sleep(2005 * time.Millisecond)
 
 	action, err = h.PostEphemeralOp(testUiHookResourceID(addr), plans.Open, nil)
 	if err != nil {
@@ -634,8 +634,7 @@ func TestUiHookEphemeralOp_progress(t *testing.T) {
 	want := `ephemeral.test_instance.foo: Opening...
 ephemeral.test_instance.foo: Still opening... [1s elapsed]
 ephemeral.test_instance.foo: Still opening... [2s elapsed]
-ephemeral.test_instance.foo: Still opening... [3s elapsed]
-ephemeral.test_instance.foo: Opening complete after 3s
+ephemeral.test_instance.foo: Opening complete after 2s
 `
 	if got := result.Stdout(); got != want {
 		t.Fatalf("unexpected output\n got: %q\nwant: %q", got, want)

internal/providers/testing/provider_mock.go

@@ -632,11 +632,6 @@ func (p *MockProvider) RenewEphemeralResource(r providers.RenewEphemeralResource
 		return resp
 	}
 
-	if p.CloseEphemeralResourceCalled {
-		resp.Diagnostics = resp.Diagnostics.Append(fmt.Errorf("CloseEphemeralResource called on %q before RenewEphemeralResource", r.TypeName))
-		return resp
-	}
-
 	p.RenewEphemeralResourceCalled = true
 	p.RenewEphemeralResourceRequest = r
 

internal/resources/ephemeral/ephemeral_resources.go

@@ -91,7 +91,14 @@ func (r *Resources) InstanceValue(addr addrs.AbsResourceInstance) (val cty.Value
 	}
 	inst, ok := insts.GetOk(addr)
 	if !ok {
-		return cty.DynamicVal, false
+		// Here we can assume that if the entire resource exists, the instance
+		// is valid because Close removes resources as a whole. Individual
+		// instances may not actually be present when checks are evaluated,
+		// because they are evaluated from instance nodes that are using "self".
+		// The way an instance gets "self" is to call GetResource which needs to
+		// compile all instances into a suitable value, so we may be missing
+		// instances which have not yet been opened.
+		return cty.DynamicVal, true
 	}
 	// If renewal has failed then we can't assume that the object is still
 	// live, but we can still return the original value regardless.

internal/terraform/context_apply_ephemeral_test.go

@@ -275,3 +275,87 @@ output "data" {
 		t.Error("CloseEphemeralResourceCalled not called")
 	}
 }
+
+func TestContext2Apply_ephemeralChecks(t *testing.T) {
+	// test the full validate-plan-apply lifecycle for ephemeral conditions
+	m := testModuleInline(t, map[string]string{
+		"main.tf": `
+variable "input" {
+  type = string
+}
+
+ephemeral "ephem_resource" "data" {
+  for_each = toset(["a", "b"])
+  lifecycle {
+    precondition {
+      condition = var.input == "ok"
+      error_message = "input not ok"
+    }
+    postcondition {
+      condition = self.value != null
+      error_message = "value is null"
+    }
+  }
+}
+
+provider "test" {
+  test_string = ephemeral.ephem_resource.data["a"].value
+}
+
+resource "test_object" "test" {
+}
+`,
+	})
+
+	ephem := &testing_provider.MockProvider{
+		GetProviderSchemaResponse: &providers.GetProviderSchemaResponse{
+			EphemeralResourceTypes: map[string]providers.Schema{
+				"ephem_resource": {
+					Block: &configschema.Block{
+						Attributes: map[string]*configschema.Attribute{
+							"value": {
+								Type:     cty.String,
+								Computed: true,
+							},
+						},
+					},
+				},
+			},
+		},
+	}
+
+	ephem.OpenEphemeralResourceFn = func(providers.OpenEphemeralResourceRequest) (resp providers.OpenEphemeralResourceResponse) {
+		resp.Result = cty.ObjectVal(map[string]cty.Value{
+			"value": cty.StringVal("test string"),
+		})
+		return resp
+	}
+
+	p := simpleMockProvider()
+
+	ctx := testContext2(t, &ContextOpts{
+		Providers: map[addrs.Provider]providers.Factory{
+			addrs.NewDefaultProvider("ephem"): testProviderFuncFixed(ephem),
+			addrs.NewDefaultProvider("test"):  testProviderFuncFixed(p),
+		},
+	})
+
+	diags := ctx.Validate(m, &ValidateOpts{})
+	assertNoDiagnostics(t, diags)
+
+	plan, diags := ctx.Plan(m, nil, &PlanOpts{
+		SetVariables: InputValues{
+			"input": &InputValue{
+				Value:      cty.StringVal("ok"),
+				SourceType: ValueFromConfig,
+			},
+		},
+	})
+	assertNoDiagnostics(t, diags)
+
+	// reset the ephemeral call flags
+	ephem.ConfigureProviderCalled = false
+
+	_, diags = ctx.Apply(plan, m, nil)
+	assertNoDiagnostics(t, diags)
+}

internal/terraform/node_resource_apply.go

@@ -5,6 +5,7 @@ package terraform
 
 import (
 	"github.com/hashicorp/terraform/internal/addrs"
+	"github.com/hashicorp/terraform/internal/dag"
 	"github.com/hashicorp/terraform/internal/tfdiags"
 )
 
@@ -51,14 +52,7 @@ func (n *nodeExpandApplyableResource) Name() string {
 
 func (n *nodeExpandApplyableResource) DynamicExpand(ctx EvalContext) (*Graph, tfdiags.Diagnostics) {
 	if n.Addr.Resource.Mode == addrs.EphemeralResourceMode {
-		// We need to expand the ephemeral resources the same as we do during
-		// planning, so we convert this into the plannable node on the fly.
-		// There doesn't seem to be any better way to handle this for now, since
-		// ephemeral resources need everything to happen the same as it would
-		// during planning.
-		return (&nodeExpandPlannableResource{
-			NodeAbstractResource: n.NodeAbstractResource,
-		}).DynamicExpand(ctx)
+		return n.dynamicExpandEphemeral(ctx)
 	}
 
 	var diags tfdiags.Diagnostics
@@ -71,3 +65,100 @@ func (n *nodeExpandApplyableResource) DynamicExpand(ctx EvalContext) (*Graph, tf
 
 	return nil, diags
 }
+
+// We need to expand the ephemeral resources mostly the same as we do during
+// planning. There a lot of options than happen during planning which aren't
+// applicable to apply however, and we have to make sure we don't re-register
+// checks which already recorded in the plan, so we create a pared down version
+// of the plan expansion here.
+func (n *nodeExpandApplyableResource) dynamicExpandEphemeral(ctx EvalContext) (*Graph, tfdiags.Diagnostics) {
+	var diags tfdiags.Diagnostics
+	var g Graph
+
+	expander := ctx.InstanceExpander()
+	moduleInstances := expander.ExpandModule(n.Addr.Module, false)
+
+	for _, module := range moduleInstances {
+		resAddr := n.Addr.Resource.Absolute(module)
+		expDiags := n.expandEphemeralResourceInstances(ctx, resAddr, &g)
+		diags = diags.Append(expDiags)
+	}
+
+	return &g, diags
+}
+
+func (n *nodeExpandApplyableResource) expandEphemeralResourceInstances(globalCtx EvalContext, resAddr addrs.AbsResource, g *Graph) tfdiags.Diagnostics {
+	var diags tfdiags.Diagnostics
+
+	// The rest of our work here needs to know which module instance it's
+	// working in, so that it can evaluate expressions in the appropriate scope.
+	moduleCtx := evalContextForModuleInstance(globalCtx, resAddr.Module)
+
+	// writeResourceState is responsible for informing the expander of what
+	// repetition mode this resource has, which allows expander.ExpandResource
+	// to work below.
+	moreDiags := n.recordResourceData(moduleCtx, resAddr)
+	diags = diags.Append(moreDiags)
+	if moreDiags.HasErrors() {
+		return diags
+	}
+
+	expander := moduleCtx.InstanceExpander()
+	instanceAddrs := expander.ExpandResource(resAddr)
+
+	instG, instDiags := n.ephemeralResourceInstanceSubgraph(resAddr, instanceAddrs)
+	if instDiags.HasErrors() {
+		diags = diags.Append(instDiags)
+		return diags
+	}
+	g.Subsume(&instG.AcyclicGraph.Graph)
+	return diags
+}
+
+func (n *nodeExpandApplyableResource) ephemeralResourceInstanceSubgraph(addr addrs.AbsResource, instanceAddrs []addrs.AbsResourceInstance) (*Graph, tfdiags.Diagnostics) {
+	var diags tfdiags.Diagnostics
+
+	concreteEphemeral := func(a *NodeAbstractResourceInstance) dag.Vertex {
+		a.Config = n.Config
+		a.ResolvedProvider = n.ResolvedProvider
+		a.Schema = n.Schema
+		a.ProvisionerSchemas = n.ProvisionerSchemas
+		a.ProviderMetas = n.ProviderMetas
+		a.dependsOn = n.dependsOn
+
+		// we still need the Plannable resource instance
+		return &NodeApplyableResourceInstance{
+			NodeAbstractResourceInstance: a,
+		}
+	}
+
+	// Start creating the steps
+	steps := []GraphTransformer{
+		// Expand the count or for_each (if present)
+		&ResourceCountTransformer{
+			Concrete:      concreteEphemeral,
+			Schema:        n.Schema,
+			Addr:          n.ResourceAddr(),
+			InstanceAddrs: instanceAddrs,
+		},
+
+		// Targeting
+		&TargetsTransformer{Targets: n.Targets},
+
+		// Connect references so ordering is correct
+		&ReferenceTransformer{},
+
+		// Make sure there is a single root
+		&RootTransformer{},
+	}
+
+	// Build the graph
+	b := &BasicGraphBuilder{
+		Steps: steps,
+		Name:  "nodeExpandApplyEphemeralResource",
+	}
+	graph, graphDiags := b.Build(addr.Module)
+	diags = diags.Append(graphDiags)
+
+	return graph, diags
+}

internal/terraform/node_resource_apply_instance.go

@@ -145,11 +145,23 @@ func (n *NodeApplyableResourceInstance) Execute(ctx EvalContext, op walkOperatio
 		return n.managedResourceExecute(ctx)
 	case addrs.DataResourceMode:
 		return n.dataResourceExecute(ctx)
+	case addrs.EphemeralResourceMode:
+		return n.ephemeralResourceExecute(ctx)
 	default:
 		panic(fmt.Errorf("unsupported resource mode %s", n.Config.Mode))
 	}
 }
 
+func (n *NodeApplyableResourceInstance) ephemeralResourceExecute(ctx EvalContext) tfdiags.Diagnostics {
+	_, diags := ephemeralResourceOpen(ctx, ephemeralResourceInput{
+		addr:           n.Addr,
+		config:         n.Config,
+		providerConfig: n.ResolvedProvider,
+	})
+
+	return diags
+}
+
 func (n *NodeApplyableResourceInstance) dataResourceExecute(ctx EvalContext) (diags tfdiags.Diagnostics) {
 	_, providerSchema, err := getProvider(ctx, n.ResolvedProvider)
 	diags = diags.Append(err)

version/VERSION

@@ -1 +1 @@
-1.10.0-beta1
+1.11.0-dev

website/docs/language/stacks/reference/tfstacks-cli.mdx

@@ -7,6 +7,10 @@ description: The terraform-stacks-cli is a command-line tool for validating, ini
 
 The `terraform-stacks-cli` is a command-line tool for validating, initializing, and testing Stack configurations.
 
+## Requirements
+
+The `terraform-stacks-cli` requires an `alpha` version of Terraform, and you must use at least version `terraform_1.10.0-alpha20241009` or higher. You can download an `alpha` version of Terraform on the [releases page](https://releases.hashicorp.com/terraform/). We recommend downloading the latest alpha version of Terraform to use the most up-to-date functionality.
+
 ## Installation
 
 To install the `terraform-stacks-cli`, you can download it directly [on the HashiCorp releases page](https://releases.hashicorp.com/tfstacks) or install it with one of the following package managers: Homebrew, Debian/Ubuntu, CentOS/RHEL, Fedora, or Amazon Linux.