Skip to content

Latest commit

 

History

History
49 lines (32 loc) · 2.03 KB

venv.md

File metadata and controls

49 lines (32 loc) · 2.03 KB
Raw

Python virtual environment in CMSSW

The CMSSW environment can be extended to act as a Python virtual environment. Assuming you are already in your CMSSW working area and have called cmsenv, execute the following commands:

scram-venv
cmsenv

Subsequently, when returning to the same CMSSW working area, just calling cmsenv will automatically activate the virtual environment.

Once the virtual environment is set up, any pip3 commands will automatically use the directories $CMSSW_BASE/venv/$SCRAM_ARCH/[bin,include,lib].

Background

The implementation of Python virtual environments in CMSSW is based on the modern Python3 specification given in PEP 405.

It relies on the use of include-system-site-packages in pyvenv.cfg to identify Python packages distributed as part of the base CMSSW release (on cvmfs) as read-only. This is accomplished via the use of sitecustomize.py when Python3 is built for CMSSW. (For more details, see the site module documentation.)

Compatibility

scram-venv works with any CMSSW version that includes Python3. Full compatibility starts with CMSSW_10_2_X.

CMSSW_10_1_X instructions

CMSSW_10_1_X has Python3 but not pip3, so some extra steps are required:

python3 -m ensurepip
python3 -m pip install --upgrade pip

Subsequently, the command python3 -m pip should be used rather than pip3.

Usage with CRAB

To add the $CMSSW_BASE/venv directory to your CRAB sandbox, include the following line in your CRAB config:

config.JobType.sendVenvFolder = True

There is a dedicated setting because symbolic links have to be preserved in the venv directory, which is not the default behavior for the CRAB sandbox.

This option must be enabled manually by the user because the venv directory can become quite large if many packages were installed. Installed Python packages are usually needed for local, offline analysis, rather than cmsRun jobs submitted with CRAB.