[DevX] Support filtering benchmark configs in on-demand workflow

[guangy10]

#7349 introduced benchmark configs which automatically detect all added supported delegates/quants/dtypes for a given model. It works well for the automated nightly benchmark jobs, however, for the on-demand jobs, given a model, it will run all supported configs for a model even if the user would like to run for target config. This PR is improving the DevX by support filtering benchmark configs for on-demand workflow

For example,

python .ci/scripts/gather_benchmark_configs.py  --models mv3,dl3 --devices samsung_galaxy_s22 --configs qnn_q8

will generate the matrix for qnn_q8 even if the models support other configs such as xnnpack_q8:

::set-output name=benchmark_configs::{"include": [{"model": "mv3", "config": "qnn_q8", "device_name": "samsung_galaxy_s22", "device_arn": "arn:aws:devicefarm:us-west-2:308535385114:devicepool:02a2cf0f-6d9b-45ee-ba1a-a086587469e6/e59f866a-30aa-4aa1-87b7-4510e5820dfa"}, {"model": "dl3", "config": "qnn_q8", "device_name": "samsung_galaxy_s22", "device_arn": "arn:aws:devicefarm:us-west-2:308535385114:devicepool:02a2cf0f-6d9b-45ee-ba1a-a086587469e6/e59f866a-30aa-4aa1-87b7-4510e5820dfa"}]}

Trigger an on-demand benchmark via GitHub UI:

Unit Tests:

test_device_pools_contains_all_devices (__main__.TestGatehrBenchmarkConfigs) ... ok
test_extract_all_configs_android (__main__.TestGatehrBenchmarkConfigs) ... ok
test_extract_all_configs_ios (__main__.TestGatehrBenchmarkConfigs) ... ok
test_gather_benchmark_configs_cli (__main__.TestGatehrBenchmarkConfigs) ... ok
test_gather_benchmark_configs_cli_specified_configs (__main__.TestGatehrBenchmarkConfigs) ... ok
test_gather_benchmark_configs_cli_specified_configs_raise (__main__.TestGatehrBenchmarkConfigs) ... ok
test_generate_compatible_configs_llama_model (__main__.TestGatehrBenchmarkConfigs) ... ok
test_generate_compatible_configs_non_genai_model (__main__.TestGatehrBenchmarkConfigs) ... ok
test_generate_compatible_configs_quantized_llama_model (__main__.TestGatehrBenchmarkConfigs) ... ok
test_generate_compatible_configs_unknown_model (__main__.TestGatehrBenchmarkConfigs) ok
test_is_valid_huggingface_model_id_valid (__main__.TestGatehrBenchmarkConfigs) ... ok
test_set_output_no_github_env (__main__.TestGatehrBenchmarkConfigs) ... ok

----------------------------------------------------------------------
Ran 12 tests in 0.177s

OK

Test jobs:

• Apple: https://github.com/pytorch/executorch/actions/runs/12839733888
• Android: https://github.com/pytorch/executorch/actions/runs/12839713009

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/7639

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

⏳ No Failures, 26 Pending

As of commit b7390c2 with merge base 74aace6 ():
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[guangy10]

@huydhn Could you help review this PR? Do you have any idea about the permission issue?

.ci/scripts/tests/test_gather_benchmark_configs.py::TestGatehrBenchmarkConfigs::test_gather_benchmark_configs_cli_specified_configs - AssertionError: 1 != 0 : Error: Traceback (most recent call last):
  File "/pytorch/executorch/.ci/scripts/gather_benchmark_configs.py", line 282, in <module>
    get_benchmark_configs()
  File "/pytorch/executorch/.ci/scripts/gather_benchmark_configs.py", line 278, in get_benchmark_configs
    set_output("benchmark_configs", json.dumps(benchmark_configs))
  File "/pytorch/executorch/.ci/scripts/gather_benchmark_configs.py", line 183, in set_output
    with open(str(os.getenv("GITHUB_OUTPUT")), "a") as env:
PermissionError: [Errno 13] Permission denied: '/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_b3721eeb-2e15-4cc1-999b-bdcdda100ae6'

[guangy10]

Write to stdout in unit tests due to permission to access GITHUB_OUTPUT

[facebook-github-bot]

@guangy10 has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

[huydhn]

Let me take a look at the PR.

For that unit test failures, let's try unset GITHUB_OUTPUT before running the test for the sake of testing. The problem is that unlike where gather_benchmark_configs script is run https://github.com/pytorch/executorch/blob/main/.github/workflows/apple-perf.yml#L86 which has access to GITHUB_OUTPUT variable under the same runner user, the unit test https://github.com/pytorch/executorch/blob/main/.github/workflows/_unittest.yml#L24 is run inside a Docker container. So, calling that from the test would be run under whatever user the container is using. You couldn't set GITHUB_OUTPUT inside a linux_job script anyway.

[guangy10]

Let me take a look at the PR.

For that unit test failures, let's try unset GITHUB_OUTPUT before running the test for the sake of testing. The problem is that unlike where gather_benchmark_configs script is run https://github.com/pytorch/executorch/blob/main/.github/workflows/apple-perf.yml#L86 which has access to GITHUB_OUTPUT variable under the same runner user, the unit test https://github.com/pytorch/executorch/blob/main/.github/workflows/_unittest.yml#L24 is run inside a Docker container. So, calling that inside the test would be run under whatever user the container is using. You couldn't set GITHUB_OUTPUT inside the linux_job script anyway.

yeah, just updated the code to write to stdout instead if raising a permission exception. Rerunning the test now

[guangy10]

The tests pass on Linux but failed on macOS runner. It seems like "::set-ouput" is not set on macOS runner.

Expected to have:

Using provided configs ['coreml_fp16', 'xnnpack_q8'] for model 'mv2'
Discovered all supported configs for model 'dl3': ['xnnpack_q8', 'coreml_fp16', 'mps']
Using provided configs ['coreml_fp16', 'xnnpack_q8'] for model 'dl3'
::set-output name=benchmark_configs::{"include": [{"model": "mv2", "config": "coreml_fp16", "device_name": "apple_iphone_15", "device_arn": "arn:aws:devicefarm:us-west-2:308535385114:devicepool:02a2cf0f-6d9b-45ee-ba1a-a086587469e6/3b5acd2e-92e2-4778-b651-7726bafe129d"}, {"model": "mv2", "config": "xnnpack_q8", "device_name": "apple_iphone_15", "device_arn": "arn:aws:devicefarm:us-west-2:308535385114:devicepool:02a2cf0f-6d9b-45ee-ba1a-a086587469e6/3b5acd2e-92e2-4778-b651-7726bafe129d"}, {"model": "dl3", "config": "coreml_fp16", "device_name": "apple_iphone_15", "device_arn": "arn:aws:devicefarm:us-west-2:308535385114:devicepool:02a2cf0f-6d9b-45ee-ba1a-a086587469e6/3b5acd2e-92e2-4778-b651-7726bafe129d"}, {"model": "dl3", "config": "xnnpack_q8", "device_name": "apple_iphone_15", "device_arn": "arn:aws:devicefarm:us-west-2:308535385114:devicepool:02a2cf0f-6d9b-45ee-ba1a-a086587469e6/3b5acd2e-92e2-4778-b651-7726bafe129d"}]}

but only got

Using provided configs ['coreml_fp16', 'xnnpack_q8'] for model 'mv2'
Discovered all supported configs for model 'dl3': ['xnnpack_q8', 'coreml_fp16', 'mps']
Using provided configs ['coreml_fp16', 'xnnpack_q8'] for model 'dl3'

And I can't reproduce it on my mac.

[huydhn]

The tests pass on Linux but failed on macOS runner. It seems like "::set-ouput" is not set on macOS runner.

I think I get it now. MacOS job doesn't run inside a Docker container, but on the host runner itself. So, the script has access to the GITHUB_OUTPUT environment variable. It passes locally because there is no such variable. I think the test need to take this into account and check both stdout and the content of GITHUB_OUTPUT (if it exists)

[guangy10]

The tests pass on Linux but failed on macOS runner. It seems like "::set-ouput" is not set on macOS runner.

I think I get it now. MacOS job doesn't run inside a Docker container, but on the host runner itself. So, the script has access to the GITHUB_OUTPUT environment variable. It passes locally because there is no such variable. I think the test need to take this into account and check both stdout and the content of GITHUB_OUTPUT (if it exists)

I see, or I can simply making it a Linux-only test, which seems better as in all CI we are running this step on Linux only

[facebook-github-bot]

@guangy10 has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

[facebook-github-bot]

@guangy10 has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

[guangy10]

@kimishpatel @digantdesai Just fyi, subscribed you to DevX related work.