Account-Customizations pipeline running out of resources for terraform apply

[bartoszbryk]

Terraform Version & Prov:

AFT Version:
1.11.1

Terraform Version & Provider Versions
Please provide the outputs of terraform version and terraform providers from within your AFT environment

terraform version

 1.3.7

terraform providers

Provider aws v5.56.0

Bug Description
We have large number of resources deployed as part of one pipeline, that utilises one state and deploys them to one account. This causes the AFT pipeline runs to fail due to lack of resources. The errors we are observing are:
The plugin.(*GRPCProvider).ValidateResourceConfig request was cancelled.
Error: Request cancelled
We would like to split the state of customizations for this particular accounts to separate state file, that's being applied in separate flow. Is this possible in the current AFT implementation? What would be the steps to achieve that

To Reproduce
Steps to reproduce the behavior:

1. Try to create large number of resources using terraform via AFT Account Customizations (>2000)
2. The terraform in the pipeline will fail due to lack of resources or timeout

Expected behavior
We would like to be able to run multiple account-customizations for the same account

[igordust]

The error message usually means that terraform is eating all the available memory. From the description you made this seems to be the case.
Currently the customizations pipeline uses an EC2 instance with 7 GB of RAM, can you try to upgrade manually to a more powerful instance in the problematic codebuild pipeline, just to understand if the issue is related to the memory as it appears to be?

Anyway, at the moment AFT doesn't give the ability to define the EC2 instance type to use for codebuild, normally an EC2 with 7GB should suffice for most of the cases, so there could be a new feature to specify the instance type for customizations pipelines.