Minimal backport of CUDA-related changes #7346

fwyzard · 2021-09-29T21:06:53Z

Update the CUDA external packaging:

include the CUDA runtime static library in the external package;
create symlinks for the library stubs to satisfy link-time checks.

Add the cuda-compatible-runtime test as a new external.

Update the CUDA external packaging: - include the CUDA runtime static library in the external package; - create symlinks for the library stubs to satisfy link-time checks. Add the cuda-compatible-runtime test as a new external.

fwyzard · 2021-09-29T21:07:01Z

please test

cmsbuild · 2021-09-29T21:07:12Z

A new Pull Request was created by @fwyzard (Andrea Bocci) for branch IB/CMSSW_12_0_X/master.

@smuzaffar, @iarspider, @mrodozov can you please review it and eventually sign? Thanks.
@perrotta, @dpiparo, @qliphy you are the release manager for this.
cms-bot commands are listed here

fwyzard · 2021-09-30T08:12:03Z

@smuzaffar @perrotta @qliphy
do you prefer to

run more extensive tests for this PR by itself, and add the changes to the SCRAM hooks separately
or
add the the changes to the SCRAM hooks together with this, and test them more extensively together
?

smuzaffar · 2021-09-30T08:16:51Z

@fwyzard , I would say, lets add the scram change (b5d1145 ) and test here extensively

cmsbuild · 2021-09-30T08:25:40Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-7d9a5b/19257/summary.html
COMMIT: 83e19fd
CMSSW: CMSSW_12_0_X_2021-09-29-1100/slc7_amd64_gcc900
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmsdist/7346/19257/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 0 differences found in the comparisons
DQMHistoTests: Total files compared: 39
DQMHistoTests: Total histograms compared: 2998564
DQMHistoTests: Total failures: 0
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 2998542
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 38 files compared)
Checked 165 log files, 37 edm output root files, 39 DQM output files
TriggerResults: no differences found

cmsbuild · 2021-09-30T08:30:22Z

Pull request #7346 was updated.

fwyzard · 2021-09-30T08:30:47Z

ok 👍
thanks for the update

fwyzard · 2021-09-30T08:30:53Z

enable gpu

fwyzard · 2021-09-30T08:31:01Z

please test

fwyzard · 2021-09-30T08:33:40Z

please test for slc7_aarch64_gcc9

fwyzard · 2021-09-30T08:34:06Z

please test for slc7_ppc64le_gcc9

fwyzard · 2021-09-30T08:34:19Z

please test for slc7_amd64_gcc10

fwyzard · 2021-09-30T08:34:33Z

please test for cc8_amd64_gcc9

fwyzard · 2021-09-30T08:35:18Z

I think we can skip the test for cc8_aarch64_gcc9 and cc8_ppc64le_gcc9.
Or should we run those as well ?

smuzaffar · 2021-09-30T09:33:13Z

I think we can skip the test for cc8_aarch64_gcc9 and cc8_ppc64le_gcc9. Or should we run those as well ?

ppc64le resources are doing nothing so I suggest to start the tests for cc8_ppc64le_gcc9 but we can skip cc8_aarch64_gcc9 as we only have one arm machine available and there is long list of pending jobs to run there

fwyzard · 2021-09-30T12:12:35Z

ok

fwyzard · 2021-09-30T12:12:44Z

please test for cc8_ppc64le_gcc9

cmsbuild · 2021-09-30T16:02:02Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-7d9a5b/19273/summary.html
COMMIT: 0171f2a
CMSSW: CMSSW_12_0_X_2021-09-29-2300/slc7_amd64_gcc900
Additional Tests: GPU
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmsdist/7346/19273/install.sh to create a dev area with all the needed externals and cmssw changes.

GPU Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 0 differences found in the comparisons
Reco comparison had 3 failed jobs
DQMHistoTests: Total files compared: 4
DQMHistoTests: Total histograms compared: 19735
DQMHistoTests: Total failures: 6
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 19729
DQMHistoTests: Total skipped: 0
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 3 files compared)
Checked 12 log files, 9 edm output root files, 4 DQM output files
TriggerResults: no differences found

Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 0 differences found in the comparisons
DQMHistoTests: Total files compared: 39
DQMHistoTests: Total histograms compared: 2998564
DQMHistoTests: Total failures: 0
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 2998542
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 38 files compared)
Checked 165 log files, 37 edm output root files, 39 DQM output files
TriggerResults: no differences found

cmsbuild · 2021-09-30T16:09:28Z

-1

Failed Tests: UnitTests
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-7d9a5b/19276/summary.html
COMMIT: 0171f2a
CMSSW: CMSSW_12_0_X_2021-09-28-2300/slc7_ppc64le_gcc9
Additional Tests: GPU
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmsdist/7346/19276/install.sh to create a dev area with all the needed externals and cmssw changes.

Unit Tests

I found errors in the following unit tests:

---> test test_PrepareInputDb had ERRORS
---> test test_MpsWorkFlow had ERRORS
---> test TestHeterogeneousCoreSonicTritonProducerGPU had ERRORS
---> test testAlignmentOfflineValidation had ERRORS
and more ...

cmsbuild · 2021-09-30T19:13:31Z

-1

Failed Tests: UnitTests
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-7d9a5b/19282/summary.html
COMMIT: 0171f2a
CMSSW: CMSSW_12_0_X_2021-09-27-2300/cc8_ppc64le_gcc9
Additional Tests: GPU
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmsdist/7346/19282/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

@cms-sw [12.0] Update UTM 0.9.1 #7337

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-7d9a5b/19282/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-7d9a5b/19282/git-merge-result

Unit Tests

I found errors in the following unit tests:

---> test test_PrepareInputDb had ERRORS
---> test test_MpsWorkFlow had ERRORS
---> test test_PixelBaryCentreTool had ERRORS
---> test testFWCoreUtilities had ERRORS
and more ...

cmsbuild · 2021-09-30T21:01:21Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-7d9a5b/19275/summary.html
COMMIT: 0171f2a
CMSSW: CMSSW_12_0_X_2021-09-29-2300/cc8_amd64_gcc9
Additional Tests: GPU
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmsdist/7346/19275/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 57020 differences found in the comparisons
DQMHistoTests: Total files compared: 39
DQMHistoTests: Total histograms compared: 2998564
DQMHistoTests: Total failures: 294215
DQMHistoTests: Total nulls: 257
DQMHistoTests: Total successes: 2704070
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: -3.872 KiB( 38 files compared)
DQMHistoSizes: changed ( 10224.0 ): -0.192 KiB SiStrip/MechanicalView
DQMHistoSizes: changed ( 136.874 ): -0.023 KiB JetMET/SUSYDQM
DQMHistoSizes: changed ( 250202.181 ): -0.410 KiB SiStrip/MechanicalView
DQMHistoSizes: changed ( 25202.0 ): 0.133 KiB SiStrip/MechanicalView
DQMHistoSizes: changed ( 312.0 ): -0.004 KiB MessageLogger/Warnings
DQMHistoSizes: changed ( 4.53 ): 0.004 KiB JetMET/SUSYDQM
DQMHistoSizes: changed ( 7.3 ): -3.380 KiB SiStrip/MechanicalView
Checked 165 log files, 37 edm output root files, 39 DQM output files
TriggerResults: found differences in 12 / 38 workflows

cmsbuild · 2021-10-01T00:50:43Z

+1

Comparison Summary

Summary:

No significant changes to the logs found
ROOTFileChecks: Some differences in event products or their sizes found
Reco comparison results: 51797 differences found in the comparisons
DQMHistoTests: Total files compared: 39
DQMHistoTests: Total histograms compared: 2998564
DQMHistoTests: Total failures: 571325
DQMHistoTests: Total nulls: 16
DQMHistoTests: Total successes: 2427201
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: -1.974 KiB( 38 files compared)
DQMHistoSizes: changed ( 10224.0 ): -0.905 KiB SiStrip/MechanicalView
DQMHistoSizes: changed ( 250202.181 ): -0.469 KiB SiStrip/MechanicalView
DQMHistoSizes: changed ( 25202.0 ): -0.596 KiB SiStrip/MechanicalView
DQMHistoSizes: changed ( 312.0 ): -0.004 KiB MessageLogger/Warnings
Checked 165 log files, 37 edm output root files, 39 DQM output files
TriggerResults: found differences in 12 / 38 workflows

smuzaffar · 2021-10-01T08:31:47Z

+externals
looks good to go in 12.0.X. Failed unit tests for non-x86-64 archs are already in the IBs.

cmsbuild · 2021-10-01T08:32:07Z

This pull request is fully signed and it will be integrated in one of the next IB/CMSSW_12_0_X/master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @perrotta, @dpiparo, @qliphy (and backports should be raised in the release meeting by the corresponding L2)

cmsbuild · 2021-10-01T09:59:06Z

-1

Failed Tests: UnitTests
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-7d9a5b/19277/summary.html
COMMIT: 0171f2a
CMSSW: CMSSW_12_0_X_2021-09-28-2300/slc7_aarch64_gcc9
Additional Tests: GPU
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmsdist/7346/19277/install.sh to create a dev area with all the needed externals and cmssw changes.

Unit Tests

I found errors in the following unit tests:

---> test TestFWCoreServicesDriver had ERRORS
---> test testUploadConditions had ERRORS

perrotta · 2021-10-01T12:40:45Z

+1

Minimal backport of CUDA-related changes

83e19fd

Update the CUDA external packaging: - include the CUDA runtime static library in the external package; - create symlinks for the library stubs to satisfy link-time checks. Add the cuda-compatible-runtime test as a new external.

cmsbuild added externals-pending orp-pending pending-signatures tests-started labels Sep 29, 2021

fwyzard changed the title ~~Minimal backport of CUDA-related changes, to support the new SCRAM hook~~ Minimal backport of CUDA-related changes Sep 29, 2021

fwyzard mentioned this pull request Sep 29, 2021

Backport CUDA updates from the 12.1.x branch #7340

Closed

cmsbuild added tests-approved and removed tests-started labels Sep 30, 2021

[12.0.X] Build rules updated to V06-03-07

0171f2a

cmsbuild added tests-pending and removed tests-approved labels Sep 30, 2021

cmsbuild added tests-started and removed tests-pending labels Sep 30, 2021

cmsbuild added tests-approved and removed tests-started labels Sep 30, 2021

cmsbuild added externals-approved fully-signed and removed externals-pending pending-signatures labels Oct 1, 2021

cmsbuild added orp-approved and removed orp-pending labels Oct 1, 2021

cmsbuild merged commit e72f37c into cms-sw:IB/CMSSW_12_0_X/master Oct 1, 2021

fwyzard deleted the IB/CMSSW_12_0_X/master_update_CUDA_hooks branch April 1, 2022 11:59

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Minimal backport of CUDA-related changes #7346

Minimal backport of CUDA-related changes #7346

fwyzard commented Sep 29, 2021

fwyzard commented Sep 29, 2021

cmsbuild commented Sep 29, 2021

fwyzard commented Sep 30, 2021

smuzaffar commented Sep 30, 2021

cmsbuild commented Sep 30, 2021

cmsbuild commented Sep 30, 2021

fwyzard commented Sep 30, 2021

fwyzard commented Sep 30, 2021

fwyzard commented Sep 30, 2021

fwyzard commented Sep 30, 2021

fwyzard commented Sep 30, 2021

fwyzard commented Sep 30, 2021

fwyzard commented Sep 30, 2021

fwyzard commented Sep 30, 2021

smuzaffar commented Sep 30, 2021

fwyzard commented Sep 30, 2021

fwyzard commented Sep 30, 2021

cmsbuild commented Sep 30, 2021

cmsbuild commented Sep 30, 2021

cmsbuild commented Sep 30, 2021

cmsbuild commented Sep 30, 2021

cmsbuild commented Oct 1, 2021

smuzaffar commented Oct 1, 2021

cmsbuild commented Oct 1, 2021

cmsbuild commented Oct 1, 2021

perrotta commented Oct 1, 2021

Minimal backport of CUDA-related changes #7346

Minimal backport of CUDA-related changes #7346

Conversation

fwyzard commented Sep 29, 2021

fwyzard commented Sep 29, 2021

cmsbuild commented Sep 29, 2021

fwyzard commented Sep 30, 2021

smuzaffar commented Sep 30, 2021

cmsbuild commented Sep 30, 2021

Comparison Summary

cmsbuild commented Sep 30, 2021

fwyzard commented Sep 30, 2021

fwyzard commented Sep 30, 2021

fwyzard commented Sep 30, 2021

fwyzard commented Sep 30, 2021

fwyzard commented Sep 30, 2021

fwyzard commented Sep 30, 2021

fwyzard commented Sep 30, 2021

fwyzard commented Sep 30, 2021

smuzaffar commented Sep 30, 2021

fwyzard commented Sep 30, 2021

fwyzard commented Sep 30, 2021

cmsbuild commented Sep 30, 2021

GPU Comparison Summary

Comparison Summary

cmsbuild commented Sep 30, 2021

Unit Tests

cmsbuild commented Sep 30, 2021

Unit Tests

cmsbuild commented Sep 30, 2021

Comparison Summary

cmsbuild commented Oct 1, 2021

Comparison Summary

smuzaffar commented Oct 1, 2021

cmsbuild commented Oct 1, 2021

cmsbuild commented Oct 1, 2021

Unit Tests

perrotta commented Oct 1, 2021