panic when external changes cause a different expansion during apply

[abaddon82]

Terraform Version

Terraform version: 1.10.1

Terraform Configuration Files

# main.tf
# variable declarations omitted

terraform {
  required_version = ">= 1.9.8"

  required_providers {
    github = {
      source  = "integrations/github"
      version = "6.4.0"
    }
  }
}

provider "github" {
  owner = var.application_owner
  app_auth {
    id = var.application_id
    installation_id = var.application_installation_id
    pem_file = file("./lsf_pk.pem")
  }
}

module "landing_zone_type_repositories" {
  source    = "./github_repo"
  for_each  = { for lzt in local.landing_zone_types : "lzt-${lzt.repository}" => lzt }
  repo_name = "lzt-${each.value.repository}"
}

# ./github_repo/main.tf
# variable declarations omitted

terraform {
  required_version = ">= 1.9.8"

  required_providers {
    github = {
      source = "integrations/github"
      version = "6.4.0"
    }
  }
}

resource "github_repository" "lzt_name" {
  name = var.repo_name
  is_template = true
  visibility = var.visibility
}

Debug Output

https://gist.github.com/abaddon82/3b0320f6cc82ee811c8a11f2d2c74480

Expected Behavior

Array of GitHub repositories should be created/configured.

Actual Behavior

Application crash.

Steps to Reproduce

1. terraform init
2. terraform apply

Additional Context

No response

References

No response

[liamcervante]

Hi @abaddon82, thanks for filing this! I think this is the same root cause as #35863, but we don't have a reliable reproduction yet. Maybe we can reproduce this using your config. A couple of questions:

• Does this happen every time you execute terraform apply?
• Are you able to share the full configuration for this, or provide a minimal working example that reproduces the crash?

Thanks!

[abaddon82]

Hi @abaddon82, thanks for filing this! I think this is the same root cause as #35863, but we don't have a reliable reproduction yet. Maybe we can reproduce this using your config. A couple of questions:

• Does this happen every time you execute terraform apply?
• Are you able to share the full configuration for this, or provide a minimal working example that reproduces the crash?

Thanks!

@liamcervante : it's part of a larger deploy, but I'll see if I can pull out the relevant stuff!

ETA: yes, it happens every time I run the workflow (this is run as part of a GitHub workflow)

[jbardin]

Hi @abaddon82,

There are a few oddities I can't explain yet in the logs, but the first thing that jumps out is the landing_zone_common_config value is erroring out with Invalid function argument. Nothing seems to depend on this, but I don't have any other theories to what's going on yet. In most cases everything else should finish applying, then you will get the full error diagnostic at the end.

Can you show a little more of the configuration that is going into the landing_zone_type_repositories module for_each expression?

[abaddon82]

Here's the relevant info:

variable "landing_zones_common_config_file" {
  description = "somethingsomething"
  type        = string
  default     = "../config/v1/standard.config.json"
}

locals {
  landing_zone_types = [
    for landing_zone_type_file in fileset(var.landing_zone_type_directory, "**/*.json") : jsondecode(file("${var.landing_zone_type_directory}/${landing_zone_type_file}"))
  ]
  landing_zone_common_config = jsondecode(file(var.landing_zones_common_config_file))
}

module "landing_zone_type_repositories" {
  source    = "./github_repo"
  for_each  = { for lzt in local.landing_zone_types : "lzt-${lzt.repository}" => lzt }
  repo_name = "lzt-${each.value.repository}"
}

The JSON document just looks like this:

{
    "repository": "containercluster"
}

[abaddon82]

An important note that I've managed to verify now: this code only causes TF to crash when run in a GitHub Actions workflow. When I run the code on my own system, it works fine.

The GitHub Actions workflow runs in a ubuntu-latest environment, while my own system runs on Windows 10. Both using Terraform version 1.10.2.

[jbardin]

@abaddon82, got it!

The failing landing_zone_common_config was the clue, your apply run in the workflow is missing some of your configuration files. This also affects landing_zone_types, but because that relies on a fileset with wildcards, the missing files don't cause an error, rather landing_zone_types just ends up empty.

This then causes a mismatch between the configuration and the plan, where module.landing_zone_type_repositories[\"lzt-containercluster\"].github_repository.lzt_name has a planned action, but when we get around to evaluating that resource, the module which contains it and the variables are all missing.

[abaddon82]

Nice work!

[jbardin]

While terraform takes a config snapshot to prevent this type of change during apply, that can't capture external files like those loaded via fileset. I don't think there will be a way to detect this until we have a mechanism for serializing more detail from the plan graph and its evaluation context.

Here's the error and stack trace to make this more searchable:

panic: value for module.landing_zone_type_repositories["lzt-containercluster"].var.repo_name was requested before it was provided
goroutine 78 [running]:
runtime/debug.Stack()
	/usr/local/go/src/runtime/debug/stack.go:26 +0x64
github.com/hashicorp/terraform/internal/logging.PanicHandler()
	/Users/jbardin/terraform/internal/logging/panic.go:84 +0x184
panic({0x1070817e0?, 0x14000992b60?})
	/usr/local/go/src/runtime/panic.go:785 +0x124
github.com/hashicorp/terraform/internal/terraform.(*Graph).walk.func1.1()
	/Users/jbardin/terraform/internal/terraform/graph.go:59 +0x36c
panic({0x1070817e0?, 0x14000992b60?})
	/usr/local/go/src/runtime/panic.go:785 +0x124
github.com/hashicorp/terraform/internal/namedvals.(*values[...]).GetExactResult(0x1077e9ba0, {{0x1400089de20, 0x1, 0x1}, {{}, {0x140006ab440, 0x9}}})
	/Users/jbardin/terraform/internal/namedvals/values.go:88 +0x180
github.com/hashicorp/terraform/internal/namedvals.(*State).GetInputVariableValue(0x1071d2b00?, {{0x1400089de20, 0x1, 0x1}, {{}, {0x140006ab440, 0x9}}})
	/Users/jbardin/terraform/internal/namedvals/state.go:47 +0xe4
github.com/hashicorp/terraform/internal/terraform.(*evaluationStateData).GetInputVariable(0x1400089a480, {{}, {0x140006ab440?, 0x0?}}, {{0x1400079abe8, 0x13}, {0xb, 0xc, 0x93}, {0xb, ...}})
	/Users/jbardin/terraform/internal/terraform/evaluate.go:294 +0x1f4
github.com/hashicorp/terraform/internal/lang.(*Scope).evalContext(0x1400089a510, {0x14000992ab0, 0x2, 0x2}, {0x0, 0x0})
	/Users/jbardin/terraform/internal/lang/eval.go:390 +0x15f8
github.com/hashicorp/terraform/internal/lang.(*Scope).EvalContext(...)
	/Users/jbardin/terraform/internal/lang/eval.go:246
github.com/hashicorp/terraform/internal/lang.(*Scope).EvalBlock(0x1400089a510, {0x1077ca4c0, 0x140009d8780}, 0x14000990780)
	/Users/jbardin/terraform/internal/lang/eval.go:59 +0x9c
github.com/hashicorp/terraform/internal/terraform.(*BuiltinEvalContext).EvaluateBlock(0x1?, {0x1077ca7f0, 0x1400037da20}, 0x14000990780, {0x0?, 0x0?}, {{{{0x0, 0x0}}, {0x0, 0x0}}, ...})
	/Users/jbardin/terraform/internal/terraform/eval_context_builtin.go:325 +0xe0
github.com/hashicorp/terraform/internal/terraform.(*NodeAbstractResourceInstance).plan(0x140005d7208, {0x1077ed978, 0x140009c4200}, 0x140009a6340, 0x0, 0x0, {0x0, 0x0, 0x0})
	/Users/jbardin/terraform/internal/terraform/node_resource_abstract_instance.go:817 +0xa4c
github.com/hashicorp/terraform/internal/terraform.(*NodeApplyableResourceInstance).managedResourceExecute(0x140008563f0, {0x1077ed978, 0x140009c4200})
	/Users/jbardin/terraform/internal/terraform/node_resource_apply_instance.go:268 +0x860
github.com/hashicorp/terraform/internal/terraform.(*NodeApplyableResourceInstance).Execute(0x1077efcc0?, {0x1077ed978?, 0x140009c4200?}, 0x20?)
	/Users/jbardin/terraform/internal/terraform/node_resource_apply_instance.go:123 +0x94