Ability to add custom post-processing of generated HCL when using terraform plan --generate-config-out

[patrickcping]

Terraform Version

`v0.9`

Use Cases

Generate Terraform configuration as part of a 'config export' pipeline/script to build a Terraform configuration package to migrate use cases from a source environment to a target environment. The generated HCL configuration package should be parameterised at point of generation, repetitive manual interventions (that may be API service specific) reduced to a minimum on each export through custom scripting.

Ideally, the resulting generated files are owned by the generator process (overwritten with environment agnostic content each time), and the admin creates separate files to fulfil needed variable dependencies

Optionally, allow for splitting the generated Terraform content (currently targeted to a single .tf file) into multiple files for readability.

Context

We encourage our customers to use import {} blocks and use the terraform plan --generate-config-out ... command to generate their HCL for a fully configured environment, of which there may be hundreds of resources, to be able to promote their configuration from source environment to target environment. We have a CLI tool that crawls our service and generates the relevant import {} blocks against a template, the aim of which is to save our customers time in generating their HCL and importing state for a manually-configured environment.

When using many import {} blocks for many resources, representing the configuration of the entire environment, the following problems are observed:

• The generated HCL from the terraform plan --generate-config-out .. is likely to immediately fail against the source environment on the final terraform plan phase (after HCL is generated), with multiple errors that requires manual correction. This can include:

  • Secret values that need to be changed from null to a variable reference in the generated HCL (which is expected behaviour from the HCL generation routine)
  • Service specific depends_on need to be added as the Terraform CLI rightly has no awareness of this requirement on certain resources
  • Service specific changes to meet custom validation requirements in the provider

• The generated HCL isn't appropriate to use against a separate target environment because the generated HCL needs sanitisation

  • Environment specific values to be variable references (such as computed resource IDs) - this could be made part of the HCL generation process but not considered in this request
  • Environment specific URLs to be made variable references

If a customer wants to "refresh" their Terraform configuration against a "gold build" environment, there is considerable manual work each time HCL is re-generated.

Attempted Solutions

None as yet, but we prefer not to invest engineering time to create our own custom HCL generator process when a Terraform standard is emerging.

Proposal

The following changes would allow us to meet the use case:

Ability to generate "Terraform JSON" (.tf.json files)

The resulting JSON from the terraform plan --generate-config-out ... command can be interpreted by standard libraries and commands (such as jq) to be able to do value replacement/file re-organisation/service specific Terraform injection

Custom post-processing CLI hook

A CLI parameter hook on the terraform plan --generate-config-out ... to provide the path to a custom post-processing script (e.g., shell script) that would be run after the HCL is generated in JSON format, but before the terraform plan activity. If the postprocessing shell script has resolved the expected plan errors, the command should complete successfully.

Example

terraform plan --generate-config-out=generated.tf --generate-config-format=json --generate-config-postprocess-file=./value_replacements_with_jq.sh

References

No response

[crw]

Thanks for this feature request! If you are viewing this issue and would like to indicate your interest, please use the 👍 reaction on the issue description to upvote this issue. We also welcome additional use case descriptions. Thanks again!