From 698a2bb92ab73ede8750f45c8743ba121cbd0de1 Mon Sep 17 00:00:00 2001 From: Hilary James Oliver Date: Wed, 13 Sep 2023 21:31:24 +1200 Subject: [PATCH] Update Jinja2 and EmPy templating sections. (#631) --- src/user-guide/writing-workflows/empy.rst | 8 ++ src/user-guide/writing-workflows/jinja2.rst | 121 ++++++++++++-------- 2 files changed, 80 insertions(+), 49 deletions(-) diff --git a/src/user-guide/writing-workflows/empy.rst b/src/user-guide/writing-workflows/empy.rst index 7d068f8fad2..c006107e21e 100644 --- a/src/user-guide/writing-workflows/empy.rst +++ b/src/user-guide/writing-workflows/empy.rst @@ -48,3 +48,11 @@ reasons:* extensions makes it easier to refactor and maintain the template code as its complexity grows. Inline Python code can be gathered into subroutines and eventually into separate modules and packages in a seamless manner. + + +Workflow files and Context Variables +------------------------------------ + +The EmPy processor has the same access as Jinja2 to +:ref:`workflow files ` and +:ref:`workflow context variables `. diff --git a/src/user-guide/writing-workflows/jinja2.rst b/src/user-guide/writing-workflows/jinja2.rst index 69bed641ccb..841aba12728 100644 --- a/src/user-guide/writing-workflows/jinja2.rst +++ b/src/user-guide/writing-workflows/jinja2.rst @@ -7,11 +7,9 @@ Jinja2 .. tutorial:: Configuration Consolidation Tutorial -Cylc supports use of the `Jinja2`_ template processor in workflow -configurations. Jinja2 variables, expressions, loop control structures, -conditional logic, etc., are processed to generate the final configuration. To -the template processor, Jinja2 code is embedded in arbitrary text, but the -result after processing must be valid Cylc syntax. +Cylc supports the `Jinja2`_ template processor in workflow configurations. +Jinja2 code can appear anywhere in the file. The result after Jinja2 processing +must be valid Cylc syntax. To use Jinja2, put a hash-bang comment in the first line of :cylc:conf:`flow.cylc`: @@ -19,22 +17,18 @@ To use Jinja2, put a hash-bang comment in the first line of :cylc:conf:`flow.cyl #!jinja2 -Template processing is the first thing done on parsing a workflow configuration -so Jinja2 can appear anywhere in the file. - Embedded Jinja2 code should be reasonably easy to understand for those with -coding experience; but if not, Jinja2 is well documented `here -`_. +coding experience; otherwise Jinja2 is documented `here `_. Uses of Jinja2 in Cylc include: - Inclusion or exclusion of config sections by logical switch, e.g. to make portable workflows - - Computation of config values from in input data + - Computation of config values from input data - Inclusion of files and sub-templates - - Loop over parameters to generate groups of similar tasks and associated + - Looping over parameters to generate groups of similar tasks and associated dependencies - but see :ref:`Parameterized Tasks ` for a - simpler alternative to this use case + simpler alternative to this where appropriate .. _fig-jinja2-ensemble: @@ -99,7 +93,7 @@ to generate task names automatically with built-in R1 = "foo => mem => post => bar" -The next example, which generates weather forecasts over a number of cities, is +The next workflow, which generates weather forecasts over a number of cities, is more complex. To add a new city and associated tasks and dependencies just add the new city name to list at the top of the file. It makes use of Jinja2 variables, loops, math, and logical flags to include or exclude tasks. @@ -120,54 +114,83 @@ variables, loops, math, and logical flags to include or exclude tasks. New York City task family expanded. -Accessing Environment Variables -------------------------------- +.. _jinja2.workflow_files: -Cylc automatically imports the environment to the template's global namespace -(see :ref:`CustomJinja2Filters`) in a dictionary called ``environ``: +Access to Workflow Files +------------------------ -.. code-block:: cylc +Your Jinja2 code can see the workflow directory by using +:ref:`Python code ` +that simply reads from the *current working directory*. - #!Jinja2 - #... - [runtime] - [[root]] - [[[environment]]] - WORKFLOW_OWNER_HOME_DIR_ON_WORKFLOW_HOST = {{environ['HOME']}} +This will be the source directory if parsing a source workflow, or the run +directory if parsing an installed workflow. -In addition, the following variables are exported to this environment -(hence are available in the ``environ`` dict) to provide workflow context: -.. code-block:: sub +.. _jinja2.workflow_context: - CYLC_VERBOSE # Verbose mode, true or false - CYLC_DEBUG # Debug mode (even more verbose), true or false +Workflow Context variables +-------------------------- - CYLC_WORKFLOW_ID # Workflow ID - CYLC_WORKFLOW_NAME # Workflow name - # (the ID with the run name removed) +Jinja2 CYLC variables available when parsing any workflow (source or installed): - CYLC_WORKFLOW_LOG_DIR # Workflow log directory. - CYLC_WORKFLOW_RUN_DIR # Location of the run directory in - # workflow host, e.g. ~/cylc-run/foo - CYLC_WORKFLOW_SHARE_DIR # Workflow (or task post parsing!) - # shared directory. - CYLC_WORKFLOW_WORK_DIR # Workflow work directory. +.. table:: -.. warning:: + ====================== ============== + CYLC_VERSION Version of Cylc parsing the configuration + CYLC_WORKFLOW_NAME Workflow name (source, or run ID minus run name) + CYLC_TEMPLATE_VARS Variables set by '--set' CLI options or plugins + ====================== ============== + + +Additional Jinja2 CYLC variables available when parsing an installed workflow: + +.. table:: - The environment is read on the workflow host when the configuration is - parsed. It is not read at run time by jobs on the job platform. + ====================== ============== + CYLC_WORKFLOW_ID Workflow ID + CYLC_WORKFLOW_RUN_DIR Workflow run directory + ====================== ============== -The following Jinja2 variables are also available (i.e. standalone, -not in the ``environ`` dict): -``CYLC_VERSION`` - Version of Cylc used. +Additional Jinja2 CYLC variables available when the scheduler is parsing an +installed workflow at run time: + +.. table:: + + ======================= ============== + CYLC_WORKFLOW_LOG_DIR Workflow log sub-directory + CYLC_WORKFLOW_SHARE_DIR Workflow share sub-directory + CYLC_WORKFLOW_WORK_DIR Workflow work sub-directory + ======================= ============== + +.. note:: + + Set default values for CYLC variables that are only defined for installed or + running workflows, to allow successful parsing in other contexts as well: + ``{{CYLC_WORKFLOW_RUN_DIR | default("not-defined")}}``. + + +Environment Variables +--------------------- + +Cylc automatically imports the parse-time environment to the template +processor's global namespace (see :ref:`CustomJinja2Filters`), +in a dictionary called ``environ``: + +.. code-block:: cylc + + #!Jinja2 + #... + [runtime] + [[root]] + [[[environment]]] + HOME_DIR_ON_WORKFLOW_HOST = {{environ['HOME']}} + +.. warning:: -``CYLC_TEMPLATE_VARS`` - All variables set by the ``-s``, ``--set-file`` or ``--set-list`` options, - or by a plugin. + The environment is read during configuration parsing. It is not the run time + job environment. .. _CustomJinja2Filters: