Scripts to copy picoDsts to HPSS in tar files on a production day basis. A sub-module of the SDMS suite.
- submitTarJobs.sh - Creates the list of 'days' to be backed up and submits jobs
- tarToHPSS.csh - Actual htar called by submitted jobs
- checkStoredFiles.sh - Run afterwards to check if all tar files have been created
THE script. It creates the list of 'days' to be backed up and submits jobs.
Important: run script as starofl
user
Select the production to be backed up via those parameters within the script:
# -- run year
runs="Run14"
# -- subset of production
production="200GeV/physics2/P15ic"
Do not change those without need!
# -- Base folder to your installation
basePath=/global/homes/s/starofl/SDMS/tarToHPSS
# --Folder on disk where picoDsts are stored before
projectBaseDir=/project/projectdirs/starprod/picodsts
# -- Folder on HPSS where picoDsts will be backed up to
hpssBaseDir=/nersc/projects/starofl/picodsts
- All log files of the jobs will be in a
log.<run>
folder. tar_done_<run>.list
andtar_done_<run>.list.fail
are created and filled bytarToHPSS.csh
according to success of HTAR operation.
After all backup jobs have been finished, run this script to check if all have been successfully backup'ed.
You can check this via: qstat -u starofl | grep tarToHPSS
For Selection of dataset and the folders, see above at submitTarJobs.sh
Executes actual HTAR
command.
Fills tar_done_<run>.list
and tar_done_<run>.list.fail
according to success of HTAR operation.
- The package needs to be installed at NERSC.
- Module needs to be run as
starofl
user !!
- Check that you are
starofl
user - Clone from git into
$HOME
- Change into
$HOME/SDMS/tarToHPSS
cd $HOME
git clone https://github.com/jthaeder/SDMS.git
cd /SDMS/tarToHPSS
- Set the proper
runs
andproduction
variable insubmitTarJobs.sh
and incheckStoredFiles.sh
- Execute
submitTarJobs.sh