Useful torque and moab commands for managing batch jobs

These commands can be found at the following paths (put these in your PATH environment variable):

/opt/torque/bin
/opt/moab/bin

mdiag -p            # show priorities for eligible jobs
mdiag -p -v         # show priorities for eligible and active jobs

showstats           # show useful stats for the whole cluster, 
                    # including the number of active/total cpu cores
showstart           # show estimated start time, based on requested job times
checkjob [jobid]    # show info on current status of job. Can be useful 
                    # working out why a job is not starting. Use -v flag 
                    # for detailed per-node info.
qstat -t            # expands array jobs

mjobctl -s [jobid]  # suspend job
mjobctl -r [jobid]  # resume job

Location of Torque log archives

Every 15 minutes the primary Torque server logs are rsync'd to mskcc-ln1 in the following paths:

/var/spool/torque/server_priv/accounting/
/var/spool/torque/server_logs/

This is useful for commands like tracejob which basically crawls those logs looking for and summarizing data from them for a particular job.

At this time the Moab logs are NOT replicated off the Moab server. (Being looked at)