This map does not have an element with the key "stdout"

[nitrocode]

Problem

When I apply my terraform module, I get the correct outputs. If I remove my .terraform directory and do another terraform init && terraform plan, I get this failure.

Error

Error: Missing map element

  on .terraform/modules/module_one.module_two.output_one/matti-terraform-shell-resource-83a3bf7/outputs.tf line 6, in output "stdout":
   6:   value = null_resource.contents.triggers == null ? null : null_resource.contents.triggers.stdout
    |----------------
    | null_resource.contents.triggers is map of string with 1 element

This map does not have an element with the key "stdout".


Error: Missing map element

  on .terraform/modules/module_one.module_two.output_one/matti-terraform-shell-resource-83a3bf7/outputs.tf line 10, in output "stderr":
  10:   value = null_resource.contents.triggers == null ? null : null_resource.contents.triggers.stderr
    |----------------
    | null_resource.contents.triggers is map of string with 1 element

This map does not have an element with the key "stderr".


Error: Missing map element

  on .terraform/modules/module_one.module_two.output_one/matti-terraform-shell-resource-83a3bf7/outputs.tf line 15, in output "exitstatus":
  15:     null_resource.contents.triggers.exitstatus == null ? null : (
    |----------------
    | null_resource.contents.triggers is map of string with 1 element

This map does not have an element with the key "exitstatus".


Releasing state lock. This may take a few moments...

If I do a terraform show | less

# module.module_one.module.module_two.module.output_one.null_resource.contents:
resource "null_resource" "contents" {
    id       = "01234567890123456789-snip"
    triggers = {
        "exitstatus" = "0"
        "id"         = "0123456789-snip"
        "stderr"     = ""
        "stdout"     = "my output"
    }
}

If I try to do this in the terraform console, it won't be able to find this module at all. I did notice that these outputs are not stored in my tfstate. Is that normal?

Reproduction steps

1. Create a terraform module that uses terraform-shell-resource
2. Create a backend.tf for s3
3. terraform apply
4. Remove state (basically replicating what a coworker will see) rm -rf .terraform
5. terraform init
6. terraform apply

Workaround:

terraform state rm module.module_one.module.module_two.module.output_one.null_resource
terraform apply

Reference:

• TF12 does not update .tfstate with trigger values from null_resource/local-exec provisioner hashicorp/terraform#23994
• External resource provider hashicorp/terraform-provider-external#5

[matti]

tbh - I don't actually know how my module works internally - so far it just has worked. It relies on terraform "bugs" that I "abuse" in a way.

Could you make a github repo with what I could try this out?

[nitrocode]

This may be a terraform bug.

$ terraform show
resource "null_resource" "contents" {
    id       = "snip"
    triggers = {
        "exitstatus" = "0"
        "id"         = "snip"
        "stderr"     = ""
        "stdout"     = "snip"
    }
}

When I do a terraform show -json the resource cannot be found in the tfstate. Only reference I see to null_resource is for each module that depends on it using .values.root_module.resources[].depends_on[] key.

Edit: For now I switched to the external data source as a workaround

[matti]

I'm a bit confused - external data sources are run every single time - I built a version of this module with them: https://github.com/matti/terraform-shell-outputs

[lorengordon]

I did some troubleshooting with @nitrocode and I think what is happening is this:

The module is writing the stdout/stderr to path.module which is in .terraform. In null_resource.contents, it is then using fileexists to check if the file exists and setting triggers.stdout to null if it does not. In tf 0.12, I believe setting a key to null like this has the effect of removing the key from the map.

outputs.tf is then checking if the triggers dictionary is null, but it is not, it still has the triggers.id key. So the false condition is triggering and it is trying to index into the stdout key, which does not exist cuz it was null'd.

This all works fine in an initial run because triggers does not exist yet, as checked in the outputs.tf expression. It fails on subsequent runs if the .terraform directory is deleted, or if it is initialized from remote state on a new system, because the fileexists expression triggers the false condition and null's the key.

[nitrocode]

Ah ok I didn't fully understand the purpose of this module and the difference between

• https://github.com/matti/terraform-shell-resource
• https://github.com/matti/terraform-shell-outputs
• and the external data source.

It was explained to me by @lorengordon from the slack chat, that your modules offer a CRUD like approach and show the output in the terraform log whereas the external data source will run every time, like you say, and swallow the output.

Anyway, it looks like she has also found out the root issue of what I was seeing. 😄

[matti]

@lorengordon (cc: @nitrocode) I just tested that:

terraform init
terraform apply
rm -rf .terraform
terraform init
terraform apply

works fine. fileexists expression is triggered only if the resource run state is removed from the state (which does not exist in .terraform) so what @lorengordon is describing does not happen unless the state is removed.

Can you guys provide me clear steps on how to repro this?

[lorengordon]

Are you testing using this project as a module, or directly as the root config? This reproduces when used as a module, slightly modified from the code in the readme:

$ cat main.tf
module "files" {
  source = "git::https://github.com/matti/terraform-shell-resource.git?ref=v1.0.3"
  command = "ls -l"
}

output "files" {
  value = module.files.stdout
}

$ terraform init && terraform apply
...
Apply complete! Resources: 3 added, 0 changed, 0 destroyed.

Outputs:

files = total 4
-rw-r--r-- 1 loren loren 172 Mar  3 14:38 main.tf
-rw-r--r-- 1 loren loren 981 Mar  3 14:41 terraform.tfstate

$ rm -rf .terraform/

$ terraform init && terraform apply
...
module.files.random_uuid.uuid: Refreshing state... [id=89deefee-ca6c-3542-a384-6d2a6bbe6fab]
module.files.null_resource.shell: Refreshing state... [id=8311687960622957747]
module.files.null_resource.contents: Refreshing state... [id=2698920844092799183]

Error: Missing map element

  on .terraform/modules/files/outputs.tf line 6, in output "stdout":
   6:   value = null_resource.contents.triggers == null ? null : null_resource.contents.triggers.stdout
    |----------------
    | null_resource.contents.triggers is map of string with 1 element

This map does not have an element with the key "stdout".


Error: Missing map element

  on .terraform/modules/files/outputs.tf line 10, in output "stderr":
  10:   value = null_resource.contents.triggers == null ? null : null_resource.contents.triggers.stderr
    |----------------
    | null_resource.contents.triggers is map of string with 1 element

This map does not have an element with the key "stderr".


Error: Missing map element

  on .terraform/modules/files/outputs.tf line 15, in output "exitstatus":
  15:     null_resource.contents.triggers.exitstatus == null ? null : (
    |----------------
    | null_resource.contents.triggers is map of string with 1 element

This map does not have an element with the key "exitstatus".

[matti]

thanks @lorengordon (cc: @nitrocode) - I think this is fixed now in v1.0.4, see a608c3f

please re-open if this does not fix it.

[matti]

(and not only think but I managed to repro with the instructions. the current tests don't test this, but it the one-off test worked + existing tests passed)

[nitrocode]

If you follow @lorengordon steps using v1.0.4, now it will not show the output, but it will complete without an error.

$ cat main.tf
locals {
  tags = {
    env   = "production"
    files = module.files.stdout
  }
}

module "files" {
  source = "git::https://github.com/matti/terraform-shell-resource.git?ref=v1.0.4"
  command = "ls -l"
}

output "tags" {
  value = local.tags
}
$ terraform init && terraform apply
... shows output of tags with file output ...
$ rm -rf .terraform
$ terraform init && terraform apply
... shows only output of env and not files ...

A test for this would be great!

[matti]

what would be even better before that is a repro repo so that we have common test ground and not just copy/paste things around.

a test for this is very welcome as a PR.

I'm investigating this now more.

[matti]

https://github.com/matti/repro-terraform-shell-resource-26

here we go, now it's easier to understand

[matti]

so the problem might be that the second apply after the delete runs like this:

➜  repro-terraform-shell-resource-26 git:(master) ✗ terraform apply -auto-approve

module.files.random_uuid.uuid: Refreshing state... [id=3c778345-693b-cf54-86ed-d1b2ba924a37]
module.files.null_resource.shell: Refreshing state... [id=2926328498930440158]
module.files.null_resource.contents: Refreshing state... [id=5911111940097178381]
module.files.null_resource.contents: Destroying... [id=5911111940097178381]
module.files.null_resource.contents: Destruction complete after 0s
module.files.null_resource.contents: Creating...
module.files.null_resource.contents: Creation complete after 0s [id=6381684079468933733]

so it destroys the contents and re-creates it... let's see

[matti]

the whole idea of this module is based on my finding that you can store stuff in terraform triggers and now the trigger re-creates because of the the protective fileexists (like @lorengordon tried to explain, but I'm just slow, sorry)

I've hit so many roadblocks with this module (reminder, this module only exists because terraform just doesn't do this natively so everything is a massssive hack) that I'm not sure if this is the one where I give up... ...well not give up, but then this module is just intended for use cases where .terraform is not deleted.

maybe there is a way to restructure the code (again, so that it leaks in some other way then)?

[nitrocode]

I can understand the frustration and appreciate you working on this.

Alternative solutions

• define an s3 bucket where file outputs can be stored and re-retrieved
• create a terraform provider that replaced this module and it was pulled into the official list of providers so it could be installed using terraform init
• submit a pr to allow us to store arbitrary values in the tfstate
• work with terraform devs to fix the issue with the triggers

The first method sounds the easiest but the most hackish way.

[matti]

define an s3 bucket where file outputs can be stored and re-retrieved

I think this is out of scope of this module

create a terraform provider

If I had the time and patience to do this, I would have done it. Somebody could do it, not me.

submit a pr to allow us to store arbitrary values in the tfstate

Can you explain more?

work with terraform devs to fix the issue with the triggers

I've opened about 40 issues in terraform repo(s)... it's kinda.. slow..

[matti]

But I found a way to fix your specific issue, but it breaks apply+apply scenario so that it never updates the output values again from the original ones.

see https://github.com/matti/terraform-shell-resource/pull/27/files

[matti]

now I think this module needs to start supporting your use case and drop support for multiple applys - it's one off / one way only. changing the commands etc won't change the outputs. cc: @nitrocode

[matti]

maybe it can be made dynamic choice - so users can select which behavior they want

[matti]

or maybe we could store the outputs twice in two resources and if the files are missing (rm -rf .terraform) then it uses the values from the other. my brain hurts already.

[matti]

heey @lorengordon don't laugh, it might work, see updated: https://github.com/matti/terraform-shell-resource/pull/27/files

[matti]

(does your brain hurt too?)

[matti]

released this as v1.0.5 - it passes apply+apply and @nitrocode case at https://github.com/matti/repro-terraform-shell-resource-26/tree/master passes

please make a PR in that repo if this still doesn't work

[nitrocode]

This is an interesting thread hashicorp/terraform-provider-external#5 . Idk if I have the ability to do it as a golang noob but it might be fun to try to create the external resource that can solve this once and for all. The guy at the bottom of the thread made https://github.com/toddnni/terraform-provider-shell and that code could be used as a template.

I don't know how to solve the issue for this module without some external output or being able to store some input into the tfstate.

[lorengordon]

My brain does hurt also. The patch looks suitably hacky, and brilliant in a squinty-sideways kind of way. Quite appropriate! 😂 I do wish terraform would just formalize some way to capture and store shell output.

[matti]

so please test, in all my testing it shows that I've deserved a 🍺

[nitrocode]

@matti the test works! nice job!

[lorengordon]

Tested, and does survive!

terraform init && terraform apply
terraform apply
rm -rf .terraform
terraform init && terraform apply

[matti]

https://travis-ci.org/matti/terraform-shell-resource/builds/657996166?utm_source=github_status&utm_medium=notification

travis fails for some reason, it passed locally. no beer.

[matti]

what.. if I run the tests in the branch it passes and when it's merged to master it fails. maybe I just delete the master branch and use this one.

[lorengordon]

Maybe your branch is out of date with master? Bit me more than once...

[matti]

okay it's 02:18am here and I still haven't had that beer. apparently I can not use git anymore (never claimed I did). let's see now..

[matti]

okay so the approach doesn't work for both cases - this hack just postpones the problem (v1.0.5 works because you tested only your case)

[lorengordon]

actually tried the sponsor link to send beer 💸 but it says there's a problem parsing the funding.yml file. so sending sad substitute virtual 🍻 instead

[lorengordon]

What is it that's not working exactly? I changed the command and re-applied and it re-ran and I got the new output.

[matti]

are you using 1.0.5 now? or just released 1.0.6?

[lorengordon]

I was testing 1.0.5. Just updated to 1.0.6 and am again getting the same error as in OP.

[matti]

because v1.0.5 doesn't pass tests (https://travis-ci.org/matti/terraform-shell-resource/builds/658000605?utm_medium=notification&utm_source=github_status)

and v1.0.6 passes, but it fails with https://github.com/matti/repro-terraform-shell-resource-26/tree/master

[matti]

hmm, is the apply+apply test faulty

[matti]

https://github.com/matti/terraform-shell-resource/blob/master/tests/trigger/test.bats#L29-L38

[matti]

I changed the command and re-applied and it re-ran and I got the new output.

But you don't get the new output if you don't delete .terraform - if you keep .terraform and just change the command it doesn't update in v1.0.5

It does run the command, but the output is not updated, because the trigger which stores the contents is not updated.

[matti]

@lorengordon @nitrocode please test with v1.0.7 - it seems to support every case combination known to humankind. also have a look at 7fffe89#diff-7a370d8342e7203b805911c92454f0f4R103 implementation. It's crazy, but works.

[matti]

also updated https://github.com/matti/repro-terraform-shell-resource-26/tree/master

[lorengordon]

But you don't get the new output if you don't delete .terraform - if you keep .terraform and just change the command it doesn't update in v1.0.5

It does run the command, but the output is not updated, because the trigger which stores the contents is not updated.

You're right! It was a false confirmation. So sorry!

v1.0.7 is doing the right thing in all cases! That patch puts us in serious hack territory too. Love it! Much appreciated! 🍺 🍺 🍺