From 13da1c047ad671c00a7b0b13cb94a84034b4b59d Mon Sep 17 00:00:00 2001 From: Hilary James Oliver Date: Tue, 25 Jul 2023 19:59:59 +1200 Subject: [PATCH 1/3] Update Jinja2 and EmPy templating sections. --- src/user-guide/writing-workflows/empy.rst | 8 ++ src/user-guide/writing-workflows/jinja2.rst | 121 ++++++++++++-------- 2 files changed, 80 insertions(+), 49 deletions(-) diff --git a/src/user-guide/writing-workflows/empy.rst b/src/user-guide/writing-workflows/empy.rst index 7d068f8fad..c006107e21 100644 --- a/src/user-guide/writing-workflows/empy.rst +++ b/src/user-guide/writing-workflows/empy.rst @@ -48,3 +48,11 @@ reasons:* extensions makes it easier to refactor and maintain the template code as its complexity grows. Inline Python code can be gathered into subroutines and eventually into separate modules and packages in a seamless manner. + + +Workflow files and Context Variables +------------------------------------ + +The EmPy processor has the same access as Jinja2 to +:ref:`workflow files ` and +:ref:`workflow context variables `. diff --git a/src/user-guide/writing-workflows/jinja2.rst b/src/user-guide/writing-workflows/jinja2.rst index 69bed641cc..f105e3b5f0 100644 --- a/src/user-guide/writing-workflows/jinja2.rst +++ b/src/user-guide/writing-workflows/jinja2.rst @@ -7,11 +7,9 @@ Jinja2 .. tutorial:: Configuration Consolidation Tutorial -Cylc supports use of the `Jinja2`_ template processor in workflow -configurations. Jinja2 variables, expressions, loop control structures, -conditional logic, etc., are processed to generate the final configuration. To -the template processor, Jinja2 code is embedded in arbitrary text, but the -result after processing must be valid Cylc syntax. +Cylc supports the `Jinja2`_ template processor in workflow configurations. +Jinja2 code can appear anywhere in the file. The result after Jinja2 processing +must be valid Cylc syntax. To use Jinja2, put a hash-bang comment in the first line of :cylc:conf:`flow.cylc`: @@ -19,22 +17,18 @@ To use Jinja2, put a hash-bang comment in the first line of :cylc:conf:`flow.cyl #!jinja2 -Template processing is the first thing done on parsing a workflow configuration -so Jinja2 can appear anywhere in the file. - Embedded Jinja2 code should be reasonably easy to understand for those with -coding experience; but if not, Jinja2 is well documented `here -`_. +coding experience; otherwise Jinja2 is documented `here `_. Uses of Jinja2 in Cylc include: - Inclusion or exclusion of config sections by logical switch, e.g. to make portable workflows - - Computation of config values from in input data + - Computation of config values from input data - Inclusion of files and sub-templates - - Loop over parameters to generate groups of similar tasks and associated + - Looping over parameters to generate groups of similar tasks and associated dependencies - but see :ref:`Parameterized Tasks ` for a - simpler alternative to this use case + simpler alternative to this where appropriate .. _fig-jinja2-ensemble: @@ -99,7 +93,7 @@ to generate task names automatically with built-in R1 = "foo => mem => post => bar" -The next example, which generates weather forecasts over a number of cities, is +The next workflow, which generates weather forecasts over a number of cities, is more complex. To add a new city and associated tasks and dependencies just add the new city name to list at the top of the file. It makes use of Jinja2 variables, loops, math, and logical flags to include or exclude tasks. @@ -120,54 +114,83 @@ variables, loops, math, and logical flags to include or exclude tasks. New York City task family expanded. -Accessing Environment Variables -------------------------------- +.. _jinja2.workflow_files: -Cylc automatically imports the environment to the template's global namespace -(see :ref:`CustomJinja2Filters`) in a dictionary called ``environ``: +Access to Workflow Files +------------------------ -.. code-block:: cylc +Your Jinja2 code can see the workflow directory by using +:ref:`Python code ` +that simply reads from the *current working directory*. - #!Jinja2 - #... - [runtime] - [[root]] - [[[environment]]] - WORKFLOW_OWNER_HOME_DIR_ON_WORKFLOW_HOST = {{environ['HOME']}} +This will be the source directory if parsing a source workflow, or the run +directory if parsing an installed workflow. -In addition, the following variables are exported to this environment -(hence are available in the ``environ`` dict) to provide workflow context: -.. code-block:: sub +.. _jinja2.workflow_context: - CYLC_VERBOSE # Verbose mode, true or false - CYLC_DEBUG # Debug mode (even more verbose), true or false +Workflow Context variables +-------------------------- - CYLC_WORKFLOW_ID # Workflow ID - CYLC_WORKFLOW_NAME # Workflow name - # (the ID with the run name removed) +Jinja2 CYLC variables available when parsing any workflow (source or installed): - CYLC_WORKFLOW_LOG_DIR # Workflow log directory. - CYLC_WORKFLOW_RUN_DIR # Location of the run directory in - # workflow host, e.g. ~/cylc-run/foo - CYLC_WORKFLOW_SHARE_DIR # Workflow (or task post parsing!) - # shared directory. - CYLC_WORKFLOW_WORK_DIR # Workflow work directory. +.. table:: -.. warning:: + ====================== ============== + CYLC_VERSION Version of Cylc parsing the configuration + CYLC_WORKFLOW_NAME Workflow name (source, or run ID minus run name) + CYLC_TEMPLATE_VARS Variables set by '--set' CLI options or plugins + ====================== ============== + + +Additional Jinja2 CYLC variables available when parsing an installed workflow: + +.. table:: - The environment is read on the workflow host when the configuration is - parsed. It is not read at run time by jobs on the job platform. + ====================== ============== + CYLC_WORKFLOW_ID Workflow ID + ====================== ============== -The following Jinja2 variables are also available (i.e. standalone, -not in the ``environ`` dict): -``CYLC_VERSION`` - Version of Cylc used. +Additional Jinja2 CYLC variables available when the scheduler is parsing an +installed workflow before starting or restarting a run: + +.. table:: + + ======================= ============== + CYLC_WORKFLOW_RUN_DIR Workflow run directory + CYLC_WORKFLOW_LOG_DIR Workflow log sub-directory + CYLC_WORKFLOW_SHARE_DIR Workflow share sub-directory + CYLC_WORKFLOW_WORK_DIR Workflow work sub-directory + ======================= ============== + +.. note:: + + Set default values for CYLC variables that are only defined for installed or + running workflows, to allow successful parsing in other contexts as well: + ``{{CYLC_WORKFLOW_RUN_DIR | default("not-defined")}}``. + + +Environment Variables +--------------------- + +Cylc automatically imports the parse-time environment to the template +processor's global namespace (see :ref:`CustomJinja2Filters`), +in a dictionary called ``environ``: + +.. code-block:: cylc + + #!Jinja2 + #... + [runtime] + [[root]] + [[[environment]]] + HOME_DIR_ON_WORKFLOW_HOST = {{environ['HOME']}} + +.. warning:: -``CYLC_TEMPLATE_VARS`` - All variables set by the ``-s``, ``--set-file`` or ``--set-list`` options, - or by a plugin. + The environment is read during configuration parsing. It is not the run time + job environment. .. _CustomJinja2Filters: From 0d2388e273008198889981fb55e90cc7d0cd6f06 Mon Sep 17 00:00:00 2001 From: Hilary James Oliver Date: Wed, 13 Sep 2023 11:50:27 +1200 Subject: [PATCH 2/3] Update src/user-guide/writing-workflows/jinja2.rst Co-authored-by: Ronnie Dutta <61982285+MetRonnie@users.noreply.github.com> --- src/user-guide/writing-workflows/jinja2.rst | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/src/user-guide/writing-workflows/jinja2.rst b/src/user-guide/writing-workflows/jinja2.rst index f105e3b5f0..7a2ea16525 100644 --- a/src/user-guide/writing-workflows/jinja2.rst +++ b/src/user-guide/writing-workflows/jinja2.rst @@ -153,12 +153,11 @@ Additional Jinja2 CYLC variables available when parsing an installed workflow: Additional Jinja2 CYLC variables available when the scheduler is parsing an -installed workflow before starting or restarting a run: +installed workflow at run time: .. table:: ======================= ============== - CYLC_WORKFLOW_RUN_DIR Workflow run directory CYLC_WORKFLOW_LOG_DIR Workflow log sub-directory CYLC_WORKFLOW_SHARE_DIR Workflow share sub-directory CYLC_WORKFLOW_WORK_DIR Workflow work sub-directory From a1a0ca8bad138af11bbabfc26c69a5e2c80d071a Mon Sep 17 00:00:00 2001 From: Hilary James Oliver Date: Wed, 13 Sep 2023 11:50:36 +1200 Subject: [PATCH 3/3] Update src/user-guide/writing-workflows/jinja2.rst Co-authored-by: Ronnie Dutta <61982285+MetRonnie@users.noreply.github.com> --- src/user-guide/writing-workflows/jinja2.rst | 1 + 1 file changed, 1 insertion(+) diff --git a/src/user-guide/writing-workflows/jinja2.rst b/src/user-guide/writing-workflows/jinja2.rst index 7a2ea16525..841aba1272 100644 --- a/src/user-guide/writing-workflows/jinja2.rst +++ b/src/user-guide/writing-workflows/jinja2.rst @@ -149,6 +149,7 @@ Additional Jinja2 CYLC variables available when parsing an installed workflow: ====================== ============== CYLC_WORKFLOW_ID Workflow ID + CYLC_WORKFLOW_RUN_DIR Workflow run directory ====================== ==============