[Run3 PromptReco] TMVA in BaseMVAValueMapProducer<pat::{Electron,Muon}> uses 11 MB memory per stream

[makortel]

From #46040 (comment)

The function reco::details::loadTMVAWeights() uses 11 MB memory per stream (1 thread/stream profile vs 4 thread/stream profile), corresponding to 87 MB in 8-thread PromptReco job.

The function is called by BaseMVAValueMapProducer<pat::Electron> and BaseMVAValueMapProducer<pat::Muon> constructors.

Avoiding the replication would reduce PromptReco memory by 76 MB. I don't know if TMVA is thread safe (i.e. if it could be easily moved to a edm::GlobalCache). If it is not, then probably some other inference engine should be considered.

[cmsbuild]

cms-bot internal usage

[cmsbuild]

A new Issue was created by @makortel.

@Dr15Jones, @antoniovilela, @makortel, @mandrenguyen, @rappoccio, @sextonkennedy, @smuzaffar can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

[makortel]

assign PhysicsTools/PatAlgos

[cmsbuild]

New categories assigned: reconstruction,xpog

@ftorrresd,@hqucms,@jfernan2,@mandrenguyen you have been requested to review this Pull request/Issue and eventually sign? Thanks

[makortel]

assign CommonTools/MVAUtils

[makortel]

assign ml

[cmsbuild]

New categories assigned: ml

@valsdav,@y19y19 you have been requested to review this Pull request/Issue and eventually sign? Thanks

[makortel]

type performance-improvements

[valsdav]

TMVA::Reader is not thread safe. There is a new interface called RBDT. https://root-forum.cern.ch/t/rdataframe-multithreading-loses-events/49338/9

I can check if we can convert these models without changes.

[makortel]

The thread had a further comment https://root-forum.cern.ch/t/rdataframe-multithreading-loses-events/49338/11

Actually looking at the code, I see that there should be a lock guard in the RReader::Compute for protecting multiple model evaluations. I will look into this why it is not thread safe.

Do you know if this was resolved? (we want to avoid locks)