search-provisioned-products fails with "Exception: No healthy provisioned product found.."

[cgswong]

Terraform Version & Prov:

Terraform: 1.9.3
AWS provider: 5.61.0

AFT Version:

(Can be found in the AFT Management Account in the SSM Parameter /aft/config/aft/version)

AFT Version: 1.13.0

Terraform Version & Provider Versions

Please provide the outputs of terraform version and terraform providers from within your AFT environment

terraform version

1.9.3

terraform providers

5.61.0

Bug Description

There is inconsistency in the successful change of OU via request using aft-account-request process. Some submissions result in success, others fail in the aft-account-request-processor check. The failure occurs in call to aft_common.account_request_framework.py at line 256, but can be further traced to aft_common.service_catalog.py. See line 109, and 121:

sc_client.search_provisioned_products(
            Filters=sc_product_search_filter, PageSize=100
        )

with the filter (line 96 in aft_common.service_catalog.py):

    sc_product_search_filter: Mapping[Literal["SearchQuery"], Sequence[str]] = {
        "SearchQuery": [
            "type:CONTROL_TOWER_ACCOUNT",
        ]

Using the filter results in the error [ERROR] Exception: No healthy provisioned product found for . In executing the similar check using AWS CLIv2, aws servicecatalog search-provisioned-products --cli-input-json file://act-search.json, the service catalog product is not found. However, removing the filter the Service Catalog product for the account is located. The Service Catalog product is of type CONTROL_TOWER_ACCOUNT and in AVAILABLE status, so there seems to be something amiss with this validation.

To Reproduce

Steps to reproduce the behavior:

1. Update ManagedOrganizationalUnit with a new value in an account request file
2. Merge PR for aft-account-request with OU change
3. Review logs for aft-account-request-processor Lambda function
4. See error, [ERROR] Exception: No healthy provisioned product found for

Expected behavior

The Service Catalog product for the account in "AVAILABLE" state, of type "CONTROL_TOWER_ACCOUNT" is located, resulting in the request to update the Service Catalog account product with an OU change.

Related Logs

{"time_stamp": "2024-10-09 13:08:14,909", "module": "aft_account_request_processor", "log_level": "ERROR", "log_message": {"FILE": "aft_account_request_processor.py", "METHOD": "lambda_handler", "EXCEPTION": "No healthy provisioned product found for "}}

Traceback (most recent call last):

File "/var/task/aft_account_request_processor.py", line 95, in lambda_handler

update_existing_account(

File "/opt/python/lib/python3.11/site-packages/aft_common/account_request_framework.py", line 256, in update_existing_account

raise Exception(

Exception: No healthy provisioned product found for

Additional context

The Service Catalog product is of type CONTROL_TOWER_ACCOUNT. The product is in AVAILABLE status. The email address matches between AFT and the SC product. Executing an AWS CLI command results in the product not being found.