Monitor your rocoto based run

Click here to view full rocoto documentation on GitHub:

https://github.com/christopherwharrop/rocoto/wiki/documentation

Use rocoto commands on the command line

Start or continue a run:

rocotorun -d /path/to/workflow/database/file -w /path/to/workflow/xml/file

Check the status of the workflow:

rocotostat -d /path/to/workflow/database/file -w /path/to/workflow/xml/file [-c YYYYMMDDCCmm,[YYYYMMDDCCmm,...]] [-t taskname,[taskname,...]] [-s] [-T]

Note: YYYYMMDDCCmm = YearMonthDayCycleMinute ...where mm/Minute is ’00’ for all cycles currently.

Check the status of a job:

rocotocheck -d /path/to/workflow/database/file -w /path/to/workflow/xml/file -c YYYYMMDDCCmm -t taskname

Force a task to run (ignores dependencies - USE CAREFULLY!):

rocotoboot -d /path/to/workflow/database/file -w /path/to/workflow/xml/file -c YYYYMMDDCCmm -t taskname

Rerun task(s):

rocotorewind -d /path/to/workflow/database/file -w /path/to/workflow/xml/file -c YYYYMMDDCCmm -t taskname

Set a task to complete (overwrites current state):

rocotocomplete -d /path/to/workflow/database/file -w /path/to/workflow/xml/file -c YYYYMMDDCCmm -t taskname

Several dates and task names may be specified in the same command by adding more -c and -t options. However, lists are not allowed.

Use rocoto viewer

A GUI was designed to assist with monitoring rocoto experiments. It can be found under the ush/rocoto folder in global-workflow.

Usage:

./rocoto_viewer.py -d /path/to/workflow/database/file -w /path/to/workflow/xml/file

Note 1: Terminal/window must be wide enough to display all experiment information columns, viewer will complain if not.

Note 2: The viewer requires the full path to the database and xml files if you are not in your EXPDIR when you invoke it.

What the viewer shows:

• First column: cycle (YYYYMMDDCCmm, YYYY=year, MM=month, DD=day, CC=cycle hour, mm=minute)
• Second column: task name (a "<" symbol indicates a group/meta-task, click "x" when meta-task is selected to expand/collapse)
• Third column: job ID from scheduler
• Fourth column: job state (QUEUED, RUNNING, SUCCEEDED, FAILED, or DEAD)
• Fifth column: exit code (0 if all ended well)
• Sixth column: number of tries/attempts to run job (0 when not yet run or just rewound, 1 when run once successfully, 2+ for multiple tries up to max try value where job is considered DEAD)
• Seventh column: job duration in seconds

How to navigate the viewer:

The rocoto viewer accepts both mouse and keyboard inputs. Click “h” for help menu and more options.

Available viewer commands:

c = get information on selected job
r = rewind (rerun) selected job, group, or cycle
R = run rocotorun
b = boot (forcibly run) selected job or group
-> = right arrow key, advance viewer forward to next cycle
<- = left arrow key, advance viewer backward to previous cycle
Q = quit/exit viewer

Advanced features:

• Select multiple tasks at once ** Click “Enter” on a task to select it, click on other tasks or use the up/down arrows to move to other tasks and click “Enter” to select them as well. ** When you next choose “r” for rewinding the pop-up window will now ask if you are sure you want to rewind all those selected tasks.
• Rewind entire group or cycle ** Group - While group/metatask is collapsed (<) click “r” to rewind whole group/metatask. ** Cycle - Use up arrow to move selector up past the first task until the entire left column is highlighted. Click “r” and the entire cycle will be rewound.