Removed mlperf.conf check in submission checker, removed equal issue … #1887

arjunsuresh · 2024-10-23T12:57:54Z

…mode check in conf files

Also does

Use model-info.json in the submission measurements directory: MLPerf inference v4.1 Postmortem item: Rename <system_desc_id>_<implementation_id>_<scenario>.json to model-info.json policies#182
Removes hardwired VERSION in loadgen
Fixes Improve the submission checker to safely exclude the invalid submissions and create a submission tarball of only valid submissions #1855
Fixes the issue of pypi loadgen wheel - mlperf.conf details are now embedded in the pypi wheel file.
Adds a Github action test for mlperf inference using loadgen whl downloaded from pypi
Added --skip-extra-accuracy-files-check option to submission checker to skip checking images folder for SDXL.

…mode check in conf files

github-actions · 2024-10-23T12:58:09Z

MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅

arjunsuresh · 2024-10-29T13:02:42Z

Test output

arjun@arjun-spr:~/inference/tools/submission$ python3 preprocess_submission.py --input=$HOME/inference_results_v4.1 --output=test --submitter=AMD
[2024-10-29 18:30:20,247 log_parser.py:59 INFO] Sucessfully loaded MLPerf log from closed/AMD/results/8xMI300X_2xEPYC-9374F/llama2-70b-99.9/Offline/accuracy/mlperf_log_detail.txt.
[2024-10-29 18:30:20,248 log_parser.py:59 INFO] Sucessfully loaded MLPerf log from closed/AMD/results/8xMI300X_2xEPYC-9374F/llama2-70b-99.9/Offline/performance/run_1/mlperf_log_detail.txt.
[2024-10-29 18:30:20,250 log_parser.py:59 INFO] Sucessfully loaded MLPerf log from closed/AMD/results/8xMI300X_2xEPYC-9374F/llama2-70b-99.9/Offline/performance/run_1/mlperf_log_detail.txt.
[2024-10-29 18:30:20,250 submission_checker.py:1385 INFO] Target latency: None, Latency: 919869139234, Scenario: Offline
[2024-10-29 18:30:20,252 log_parser.py:59 INFO] Sucessfully loaded MLPerf log from closed/AMD/results/8xMI300X_2xEPYC-9374F/llama2-70b-99.9/Server/accuracy/mlperf_log_detail.txt.
[2024-10-29 18:30:20,253 log_parser.py:59 INFO] Sucessfully loaded MLPerf log from closed/AMD/results/8xMI300X_2xEPYC-9374F/llama2-70b-99.9/Server/performance/run_1/mlperf_log_detail.txt.
[2024-10-29 18:30:20,255 log_parser.py:59 INFO] Sucessfully loaded MLPerf log from closed/AMD/results/8xMI300X_2xEPYC-9374F/llama2-70b-99.9/Server/performance/run_1/mlperf_log_detail.txt.
[2024-10-29 18:30:20,255 submission_checker.py:1366 INFO] Target latency: 20000000000, Early Stopping Latency: 0, Scenario: Server
[2024-10-29 18:30:20,257 log_parser.py:59 INFO] Sucessfully loaded MLPerf log from closed/AMD/results/8xMI300X_2xEPYC-9374F/llama2-70b-99/Offline/accuracy/mlperf_log_detail.txt.
[2024-10-29 18:30:20,258 log_parser.py:59 INFO] Sucessfully loaded MLPerf log from closed/AMD/results/8xMI300X_2xEPYC-9374F/llama2-70b-99/Offline/performance/run_1/mlperf_log_detail.txt.
[2024-10-29 18:30:20,259 log_parser.py:59 INFO] Sucessfully loaded MLPerf log from closed/AMD/results/8xMI300X_2xEPYC-9374F/llama2-70b-99/Offline/performance/run_1/mlperf_log_detail.txt.
[2024-10-29 18:30:20,259 submission_checker.py:1385 INFO] Target latency: None, Latency: 919869139234, Scenario: Offline
[2024-10-29 18:30:20,261 log_parser.py:59 INFO] Sucessfully loaded MLPerf log from closed/AMD/results/8xMI300X_2xEPYC-9374F/llama2-70b-99/Server/accuracy/mlperf_log_detail.txt.
[2024-10-29 18:30:20,262 log_parser.py:59 INFO] Sucessfully loaded MLPerf log from closed/AMD/results/8xMI300X_2xEPYC-9374F/llama2-70b-99/Server/performance/run_1/mlperf_log_detail.txt.
[2024-10-29 18:30:20,264 log_parser.py:59 INFO] Sucessfully loaded MLPerf log from closed/AMD/results/8xMI300X_2xEPYC-9374F/llama2-70b-99/Server/performance/run_1/mlperf_log_detail.txt.
[2024-10-29 18:30:20,264 submission_checker.py:1366 INFO] Target latency: 20000000000, Early Stopping Latency: 0, Scenario: Server
[2024-10-29 18:30:20,265 log_parser.py:59 INFO] Sucessfully loaded MLPerf log from closed/AMD/results/1xMI300X_2xEPYC-9374F/llama2-70b-99.9/Offline/accuracy/mlperf_log_detail.txt.
[2024-10-29 18:30:20,267 log_parser.py:59 INFO] Sucessfully loaded MLPerf log from closed/AMD/results/1xMI300X_2xEPYC-9374F/llama2-70b-99.9/Offline/performance/run_1/mlperf_log_detail.txt.
[2024-10-29 18:30:20,268 log_parser.py:59 INFO] Sucessfully loaded MLPerf log from closed/AMD/results/1xMI300X_2xEPYC-9374F/llama2-70b-99.9/Offline/performance/run_1/mlperf_log_detail.txt.
[2024-10-29 18:30:20,268 submission_checker.py:1385 INFO] Target latency: None, Latency: 2352997093546, Scenario: Offline
[2024-10-29 18:30:20,268 submission_checker.py:2752 ERROR] closed/AMD/compliance/1xMI300X_2xEPYC-9374F/llama2-70b-99.9/Offline/TEST06/verify_accuracy.txt is missing in closed/AMD/compliance/1xMI300X_2xEPYC-9374F/llama2-70b-99.9/Offline/TEST06
[2024-10-29 18:30:20,285 preprocess_submission.py:282 WARNING] Offline scenario result is invalid for 1xMI300X_2xEPYC-9374F: llama2-70b-99.9 in closed division. Accuracy: True, Performance: True. Compliance: False. Moving llama2-70b-99.9 results to open...
[2024-10-29 18:30:20,287 log_parser.py:59 INFO] Sucessfully loaded MLPerf log from closed/AMD/results/1xMI300X_2xEPYC-9374F/llama2-70b-99/Offline/accuracy/mlperf_log_detail.txt.
[2024-10-29 18:30:20,288 log_parser.py:59 INFO] Sucessfully loaded MLPerf log from closed/AMD/results/1xMI300X_2xEPYC-9374F/llama2-70b-99/Offline/performance/run_1/mlperf_log_detail.txt.
[2024-10-29 18:30:20,289 log_parser.py:59 INFO] Sucessfully loaded MLPerf log from closed/AMD/results/1xMI300X_2xEPYC-9374F/llama2-70b-99/Offline/performance/run_1/mlperf_log_detail.txt.
[2024-10-29 18:30:20,289 submission_checker.py:1385 INFO] Target latency: None, Latency: 2352997093546, Scenario: Offline
[2024-10-29 18:30:20,291 log_parser.py:59 INFO] Sucessfully loaded MLPerf log from closed/AMD/results/1xMI300X_2xEPYC-9374F/llama2-70b-99/Server/accuracy/mlperf_log_detail.txt.
[2024-10-29 18:30:20,292 log_parser.py:59 INFO] Sucessfully loaded MLPerf log from closed/AMD/results/1xMI300X_2xEPYC-9374F/llama2-70b-99/Server/performance/run_1/mlperf_log_detail.txt.
[2024-10-29 18:30:20,293 log_parser.py:59 INFO] Sucessfully loaded MLPerf log from closed/AMD/results/1xMI300X_2xEPYC-9374F/llama2-70b-99/Server/performance/run_1/mlperf_log_detail.txt.
[2024-10-29 18:30:20,294 submission_checker.py:1366 INFO] Target latency: 20000000000, Early Stopping Latency: 0, Scenario: Server
[2024-10-29 18:30:20,295 log_parser.py:59 INFO] Sucessfully loaded MLPerf log from closed/AMD/results/8xMI300X_2xEPYC-TURIN/llama2-70b-99.9/Offline/accuracy/mlperf_log_detail.txt.
[2024-10-29 18:30:20,297 log_parser.py:59 INFO] Sucessfully loaded MLPerf log from closed/AMD/results/8xMI300X_2xEPYC-TURIN/llama2-70b-99.9/Offline/performance/run_1/mlperf_log_detail.txt.
[2024-10-29 18:30:20,298 log_parser.py:59 INFO] Sucessfully loaded MLPerf log from closed/AMD/results/8xMI300X_2xEPYC-TURIN/llama2-70b-99.9/Offline/performance/run_1/mlperf_log_detail.txt.
[2024-10-29 18:30:20,298 submission_checker.py:1385 INFO] Target latency: None, Latency: 897168170178, Scenario: Offline
[2024-10-29 18:30:20,299 log_parser.py:59 INFO] Sucessfully loaded MLPerf log from closed/AMD/results/8xMI300X_2xEPYC-TURIN/llama2-70b-99.9/Server/accuracy/mlperf_log_detail.txt.
[2024-10-29 18:30:20,301 log_parser.py:59 INFO] Sucessfully loaded MLPerf log from closed/AMD/results/8xMI300X_2xEPYC-TURIN/llama2-70b-99.9/Server/performance/run_1/mlperf_log_detail.txt.
[2024-10-29 18:30:20,302 log_parser.py:59 INFO] Sucessfully loaded MLPerf log from closed/AMD/results/8xMI300X_2xEPYC-TURIN/llama2-70b-99.9/Server/performance/run_1/mlperf_log_detail.txt.
[2024-10-29 18:30:20,302 submission_checker.py:1366 INFO] Target latency: 20000000000, Early Stopping Latency: 0, Scenario: Server
[2024-10-29 18:30:20,304 log_parser.py:59 INFO] Sucessfully loaded MLPerf log from closed/AMD/results/8xMI300X_2xEPYC-TURIN/llama2-70b-99/Offline/accuracy/mlperf_log_detail.txt.
[2024-10-29 18:30:20,305 log_parser.py:59 INFO] Sucessfully loaded MLPerf log from closed/AMD/results/8xMI300X_2xEPYC-TURIN/llama2-70b-99/Offline/performance/run_1/mlperf_log_detail.txt.
[2024-10-29 18:30:20,306 log_parser.py:59 INFO] Sucessfully loaded MLPerf log from closed/AMD/results/8xMI300X_2xEPYC-TURIN/llama2-70b-99/Offline/performance/run_1/mlperf_log_detail.txt.
[2024-10-29 18:30:20,306 submission_checker.py:1385 INFO] Target latency: None, Latency: 897168170178, Scenario: Offline
[2024-10-29 18:30:20,308 log_parser.py:59 INFO] Sucessfully loaded MLPerf log from closed/AMD/results/8xMI300X_2xEPYC-TURIN/llama2-70b-99/Server/accuracy/mlperf_log_detail.txt.
[2024-10-29 18:30:20,308 preprocess_submission.py:176 WARNING] [Errno 2] No such file or directory: 'closed/AMD/results/8xMI300X_2xEPYC-TURIN/llama2-70b-99/Server/performance/run_1/mlperf_log_detail.txt'
[2024-10-29 18:30:20,308 preprocess_submission.py:261 WARNING] Server scenario result is invalid for 8xMI300X_2xEPYC-TURIN: llama2-70b-99 in closed and open divisions. Accuracy: True, Performance: False. Removing it...
[2024-10-29 18:30:20,321 preprocess_submission.py:284 WARNING] Server scenario result is invalid for 8xMI300X_2xEPYC-TURIN: llama2-70b-99 in closed division. Accuracy: True, Performance: False. Compliance: True. Moving other scenario results of llama2-70b-99 to open...
[2024-10-29 18:30:20,322 preprocess_submission.py:473 INFO] Division closed, submitter AMD, system 8xMI300X_2xEPYC-TURIN:                                             copying llama2-70b-99.9 results to llama2-70b-99
[2024-10-29 18:30:20,323 preprocess_submission.py:473 INFO] Division closed, submitter AMD, system 8xMI300X_2xEPYC-TURIN:                                             copying llama2-70b-99.9 results to llama2-70b-99
[2024-10-29 18:30:20,324 preprocess_submission.py:473 INFO] Division closed, submitter AMD, system 8xMI300X_2xEPYC-TURIN:                                             copying llama2-70b-99.9 results to llama2-70b-99

…ug output

…om v4.0

mrmhodak · 2024-10-29T16:41:53Z

@pgmpablo157321 to take a look

pgmpablo157321

Tested Loadgen built, demos run and submission checker. No issues, LGTM

arjunsuresh · 2024-10-30T23:57:51Z

Thank you @pgmpablo157321 for checking.

Removed mlperf.conf check in submission checker, removed equal issue …

4e1ba93

…mode check in conf files

arjunsuresh requested a review from a team as a code owner October 23, 2024 12:57

Fix loadgen wheel package

d85fa9e

arjunsuresh force-pushed the dev branch from e5d94ca to d85fa9e Compare October 23, 2024 13:44

Embed mlperf.conf as a binary string in the distribution

de458a1

arjunsuresh force-pushed the dev branch from 8ea261f to de458a1 Compare October 23, 2024 17:27

Fix xxd command for loadgen

1140549

arjunsuresh force-pushed the dev branch from f7253c8 to 1140549 Compare October 23, 2024 17:33

Fix mlperf_conf.h include

13c47f6

arjunsuresh force-pushed the dev branch from 4d6770e to 13c47f6 Compare October 23, 2024 17:49

Fix mlperf_conf.h include

e500060

arjunsuresh force-pushed the dev branch from 716441d to e500060 Compare October 23, 2024 18:46

Fix mlperf_conf.h include

9588355

arjunsuresh force-pushed the dev branch from 010b5fa to 9588355 Compare October 23, 2024 18:58

Fix mlperf_conf.h include

f26302e

arjunsuresh force-pushed the dev branch from 3b9b011 to f26302e Compare October 23, 2024 19:44

Fix mlperf_conf.h include

f506644

arjunsuresh force-pushed the dev branch from 33ffdae to f506644 Compare October 23, 2024 19:54

Fix loadgen VERSION to include patch number, remove hardcoded versions

95c5541

arjunsuresh force-pushed the dev branch from f6d0c6c to 95c5541 Compare October 23, 2024 20:03

Added vim-common deps for loadgen build on Linux gh action

5d8838f

arjunsuresh force-pushed the dev branch from 34503e8 to 5d8838f Compare October 23, 2024 20:12

Support mlperf_loadgen name

e32a3a7

arjunsuresh force-pushed the dev branch from 651f7cd to e32a3a7 Compare October 23, 2024 20:29

Added VERSION.txt src dependency in cmake

fba2222

arjunsuresh force-pushed the dev branch from cfc6045 to fba2222 Compare October 23, 2024 20:35

Added VERSION.txt src dependency in cmake

2c6641c

arjunsuresh force-pushed the dev branch from c79553d to 2c6641c Compare October 23, 2024 20:42

Fix xxd deps for loadgen

5b2ebfa

arjunsuresh temporarily deployed to release October 24, 2024 15:17 — with GitHub Actions Inactive

arjunsuresh and others added 7 commits October 25, 2024 14:29

Update submission_checker.py | replace <sut>.json by model-info.json

1cd31f9

[Automated Commit] Format Codebase

f63f304

Update submission_checker.py

99f9c7f

Retain backward compatibility of the submission checker since v4.0

6ec9013

[Automated Commit] Format Codebase

54072ed

Support impl in model-info.json

621b6dd

Update submission_checker.py

4be1d16

arjunsuresh added the postmortem 4.1 label Oct 25, 2024

arjunsuresh and others added 8 commits October 29, 2024 01:02

Submission checker cleanup, support move/removal of invalid results

6475ac9

Cleanup/fix of preprocess_submission code

85ab9f6

Fix params for get_performance_metric (submission checker)

002a836

Fix params for check_performance_dir (submission checker)

b0142f6

[Automated Commit] Format Codebase

6e8f527

Merge branch 'master' into dev

443aae0

Increment version to 4.1.23

90029be

[Automated Commit] Format Codebase

beb5c9f

arjunsuresh added 5 commits October 29, 2024 18:33

Added compliance checks to preprocess submission, use logging for deb…

72b5954

…ug output

Update test-submission-checker.yml to check inference_results_v4.1 fr…

f5bf1bf

…om v4.0

Added skip-accuarcy-files-check option to submission checker

2d537ed

Added skip-accuarcy-files-check option to submission checker

20c8d17

Added skip-accuarcy-files-check option to submission checker

ee535b4

Merge branch 'master' into dev

15628ed

pgmpablo157321 approved these changes Oct 30, 2024

View reviewed changes

Merge branch 'master' into dev

e8620ea

arjunsuresh merged commit c8c1e61 into master Oct 31, 2024
17 checks passed

github-actions bot locked and limited conversation to collaborators Oct 31, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Removed mlperf.conf check in submission checker, removed equal issue … #1887

Removed mlperf.conf check in submission checker, removed equal issue … #1887

arjunsuresh commented Oct 23, 2024 •

edited

Loading

github-actions bot commented Oct 23, 2024 •

edited

Loading

arjunsuresh commented Oct 29, 2024

mrmhodak commented Oct 29, 2024

pgmpablo157321 left a comment

arjunsuresh commented Oct 30, 2024

Removed mlperf.conf check in submission checker, removed equal issue … #1887

Removed mlperf.conf check in submission checker, removed equal issue … #1887

Conversation

arjunsuresh commented Oct 23, 2024 • edited Loading

github-actions bot commented Oct 23, 2024 • edited Loading

arjunsuresh commented Oct 29, 2024

Test output

mrmhodak commented Oct 29, 2024

pgmpablo157321 left a comment

Choose a reason for hiding this comment

arjunsuresh commented Oct 30, 2024

arjunsuresh commented Oct 23, 2024 •

edited

Loading

github-actions bot commented Oct 23, 2024 •

edited

Loading