[Bug] Discrepancies between TC list generated by PICS tool and TC list generated by TH

[jrhees-cae]

Describe the bug

See rollup doc: https://docs.google.com/document/d/11AgU-x_ffznlzlSp_J7GV4YbClkDEA2w7qvRnXQ_Jqg/edit?usp=sharing

Steps to reproduce the behavior

No response

Expected behavior

No response

Log files

No response

PICS file

PICS.zip

Screenshots

No response

Environment

No response

Additional Information

No response

[rquidute]

From an initial analysis, sharing some findings:

TC listed in TH and not in PICS Tools

TC-BRBINFO-4.1
TC-CC-2.2
TC-CCTRL-2.1
TC-CCTRL-2.2
TC-CCTRL-2.3
TC-ECOINFO-2.1
TC-ECOINFO-2.2
TC-LVL-2.3
TC-MCORE-FS.1.1
TC-MCORE-FS.1.2
TC-MCORE-FS.1.3
TC-MCORE-FS.1.4
TC-MCORE-FS.1.5
TC-RVCOPSTATE-2.1
TC-SC-2.2
TC-SC-8.1
TC-SC-8.2
TC-SC-8.3
TC-SC-8.4
TC-SC-8.5
TC-SC-8.6
TC-SC-8.7
TC-SEAR-1.2
TC-SEAR-1.3
TC-SEAR-1.4
TC-SEAR-1.5
TC-SEAR-1.6
TC-SWTCH-2.2
TC-SWTCH-2.3
TC-SWTCH-2.4
TC-SWTCH-2.5
TC-SWTCH-2.6

**All the tests above does have PICS definition in script, this is why TH is adding to the list**

Under investigation - TC listed in PICS Tools and not in TH

TC-CADMIN- 1.19
TC-CADMIN- 1.22
TC-CADMIN- 1.24
TC-DGTHREAD- 2.4
TC-IDM-9.1 - Test not found
TC-PS-2.3

Under investigation - TC listed in TH and not in PICS Tools

TC-DGSW-2.1
TC-DGSW-2.3

[jrhees-cae]

@rquidute You say 'All the tests above does have PICS definition in script, this is why TH is adding to the list'

What does this mean? Why would, for example, the TC-SWTCH-xx tests appear in the TH if there is no Switch-related PICS loaded?

[rquidute]

Hi @jrhees-cae , The TC-SWTCH-xx test cases - for instance, does not have any PICS definition inside script file, so the current implementation of TH is considering such cases that the test cases should be in the execution list.

[jrhees-cae]

I see, so they would show up as required for any DUT? That's definitely a problem -- especially if you wanted to use 'Certification Mode'

[fabiowmm]

@jrhees-cae in fact, they (PICS-less test cases) would be preselected by TH in dev mode (and the user is free to deselect) - that has always been the agreed default behavior. PICS values in the scripts are for exclusion (if the PICS is absent from the PICS file) rather than inclusion.

As for certification mode, this is a work in progress. The desired behavior is for TH to launch ALL test cases when in cert mode and the TCs themselves figure out (by querying the DUT) if they should run. @cecille can add more context here.

[rquidute]

Sharing the findings after a detailed analysis:

• The already mentioned Script files that does not contains top level PICS definition.
• TC-DGTHREAD- 2.4: Yaml Script contains DGTHREAD.S and DGTHREAD.S.F01(ERRCNT) top level PICS definition and the expected were DGTHREAD.S and DGTHREAD.S.F01.
• TC-IDM-9.1: This test file was not found. Maybe this TC is not implemented yet?
• TC-PS-2.3: Python Script contains PWRTL.S top level PICS definition and the expected was PS.S.
• There are 2 TC-DGSW-2.1 test:
  python_testing/TC_DGSW_2_1.py
app/tests/suites/certification/Test_TC_DGSW_2_1.yaml