Run PR gpu utests/relvals on both CUDA and ROCm GPUs

[iarspider]

Additional changes :

• PR Unit test job (https://cmssdt.cern.ch/jenkins/job/ib-run-pr-unittests) - add GPU_FLAVOR parameter, select node based on it.

• PR Relval job (https://cmssdt.cern.ch/jenkins/job/ib-run-pr-relvals/) - select node based on TEST_FLAVOR

• Monitoring job (https://cmssdt.cern.ch/jenkins/job/jenkins-monitor-queue/) needs to be changed - kill pr test jobs that are stuck due to lumi node being offline

• If ROCm job is killed due to node being offline, mark commit status!

[cmsbuild]

cms-bot internal usage

[iarspider]

please test with cms-sw/cmssw#46579

to check that cpu tests are not broken

[cmsbuild]

-1

Failed Tests: ClangBuild
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-b157ff/43916/summary.html
COMMIT: 118dd7e
CMSSW: CMSSW_15_0_X_2025-01-22-1100/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cms-bot/2418/43916/install.sh to create a dev area with all the needed externals and cmssw changes.

CMS deprecated warnings: 1 CMS deprecated warnings found, see summary page for details.

Clang Build

I found compilation warning while trying to compile with clang. Command used:

USER_CUDA_FLAGS='--expt-relaxed-constexpr' USER_CXXFLAGS='-Wno-register -fsyntax-only' /usr/bin/time -v scram build -k -j 32 COMPILER='llvm compile'

See details on the summary page.

[iarspider]

please test with cms-sw/cmssw#47163

[cmsbuild]

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-b157ff/43917/summary.html
COMMIT: 118dd7e
CMSSW: CMSSW_15_0_X_2025-01-22-1100/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cms-bot/2418/43917/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

• You potentially added 1 lines to the logs
• Reco comparison results: 1664 differences found in the comparisons
• DQMHistoTests: Total files compared: 49
• DQMHistoTests: Total histograms compared: 3819085
• DQMHistoTests: Total failures: 149
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 3818916
• DQMHistoTests: Total skipped: 20
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 48 files compared)
• Checked 214 log files, 184 edm output root files, 49 DQM output files
• TriggerResults: found differences in 1 / 47 workflows

[iarspider]

test parameters:

• enable = gpu
• workflows_gpu = 141.044406,141.044408,141.044412,141.044414,141.044422,141.044424,160.03502

[iarspider]

please test

[cmsbuild]

Pull request #2418 was updated.

[iarspider]

please test

[cmsbuild]

Pull request #2418 was updated.

[cmsbuild]

Pull request #2418 was updated.

[cmsbuild]

Pull request #2418 was updated.

[iarspider]

unhold

[iarspider]

please test

[cmsbuild]

Pull request #2418 was updated.

[cmsbuild]

Pull request #2418 was updated.

[iarspider]

please test

[iarspider]

please test

[cmsbuild]

Pull request #2418 was updated.

[cmsbuild]

-1

Failed Tests: RelVals-ROCM
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-b157ff/44268/summary.html
COMMIT: 4e74ecc
CMSSW: CMSSW_15_0_X_2025-02-07-1100/el8_amd64_gcc12
Additional Tests: GPU
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cms-bot/2418/44268/install.sh to create a dev area with all the needed externals and cmssw changes.

RelVals-ROCM

• 141.044406141.044406_Run3-2023_JetMET2023D_RecoPixelOnlyTripletsGPU/step3_Run3-2023_JetMET2023D_RecoPixelOnlyTripletsGPU.log
• 12834.40612834.406_TTbar_14TeV+2024_Patatrack_PixelOnlyTripletsAlpaka/step3_TTbar_14TeV+2024_Patatrack_PixelOnlyTripletsAlpaka.log
• 141.044408141.044408_Run3-2023_JetMET2023D_RecoPixelOnlyTripletsGPU_Profiling/step3_Run3-2023_JetMET2023D_RecoPixelOnlyTripletsGPU_Profiling.log

Expand to see more relval errors ...

• 12834.403
• 12834.402

Comparison Summary

Summary:

• You potentially added 1 lines to the logs
• Reco comparison results: 4 differences found in the comparisons
• DQMHistoTests: Total files compared: 50
• DQMHistoTests: Total histograms compared: 4016960
• DQMHistoTests: Total failures: 3
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 4016937
• DQMHistoTests: Total skipped: 20
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 49 files compared)
• Checked 218 log files, 189 edm output root files, 50 DQM output files
• TriggerResults: no differences found

CUDA Comparison Summary

Summary:

• No significant changes to the logs found
• Reco comparison results: 0 differences found in the comparisons
• DQMHistoTests: Total files compared: 1
• DQMHistoTests: Total histograms compared: 0
• DQMHistoTests: Total failures: 0
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 0
• DQMHistoTests: Total skipped: 0
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0 KiB( 0 files compared)
• Checked 0 log files, 0 edm output root files, 1 DQM output files

[cmsbuild]

Pull request #2418 was updated.