diff --git a/_config.yml b/_config.yml index b2785dfa..d6c69a82 100644 --- a/_config.yml +++ b/_config.yml @@ -81,11 +81,12 @@ episode_order: - 11-connecting - 12-cluster - 13-scheduler - - 14-modules - - 15-transferring-files - - 16-parallel - - 17-resources - - 18-responsibility + - 14-environment-variables + - 15-modules + - 16-transferring-files + - 17-parallel + - 18-resources + - 19-responsibility #------------------------------------------------------------ # Values for this lesson diff --git a/_episodes/13-scheduler.md b/_episodes/13-scheduler.md index b75d1c0d..22263304 100644 --- a/_episodes/13-scheduler.md +++ b/_episodes/13-scheduler.md @@ -235,8 +235,6 @@ later episode of this lesson. > {: .solution} {: .challenge} -{% include {{ site.snippets }}/scheduler/print-sched-variables.snip %} - Resource requests are typically binding. If you exceed them, your job will be killed. Let's use wall time as an example. We will request 1 minute of wall time, and attempt to run a job for two minutes. diff --git a/_episodes/14-environment-variables.md b/_episodes/14-environment-variables.md new file mode 100644 index 00000000..b13cc5ca --- /dev/null +++ b/_episodes/14-environment-variables.md @@ -0,0 +1,258 @@ +--- +title: Environment Variables +teaching: 10 +exercises: 5 +questions: +- "How are variables set and accessed in the Unix shell?" +- "How can I use variables to change how a program runs?" +objectives: +- "Understand how variables are implemented in the shell" +- "Read the value of an existing variable" +- "Create new variables and change their values" +- "Change the behaviour of a program using an environment variable" +- "Explain how the shell uses the `PATH` variable to search for executables" +keypoints: +- "Shell variables are by default treated as strings" +- "Variables are assigned using \"`=`\" and recalled using the variable's name prefixed by \"`$`\"" +- "Use \"`export`\" to make an variable available to other programs" +- "The `PATH` variable defines the shell's search path" +--- + +> ## Episode provenance +> +> This episode has been remixed from the +> [Shell Extras episode on Shell Variables](https://github.com/carpentries-incubator/shell-extras/blob/gh-pages/_episodes/08-environment-variables.md) +> and the [HPC Shell episode on scripts](https://github.com/hpc-carpentry/hpc-shell/blob/gh-pages/_episodes/05-scripts.md) +{: .callout} + +The shell is just a program, and like other programs, it has variables. +Those variables control its execution, +so by changing their values +you can change how the shell behaves (and with a little more effort how other +programs behave). + +Variables +are a great way of saving information under a name you can access later. In +programming languages like Python and R, variables can store pretty much +anything you can think of. In the shell, they usually just store text. The best +way to understand how they work is to see them in action. + +Let's start by running the command `set` and looking at some of the variables +in a typical shell session: + +~~~ +$ set +~~~ +{: .language-bash} + +~~~ +COMPUTERNAME=TURING +HOME=/home/vlad +HOSTNAME=TURING +HOSTTYPE=i686 +NUMBER_OF_PROCESSORS=4 +PATH=/Users/vlad/bin:/usr/local/git/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin +PWD=/home/vlad +UID=1000 +USERNAME=vlad +... +~~~ +{: .output} + +As you can see, there are quite a few — in fact, +four or five times more than what's shown here. +And yes, using `set` to *show* things might seem a little strange, +even for Unix, but if you don't give it any arguments, +it might as well show you things you *could* set. + +Every variable has a name. +All shell variables' values are strings, +even those (like `UID`) that look like numbers. +It's up to programs to convert these strings to other types when necessary. +For example, if a program wanted to find out how many processors the computer +had, it would convert the value of the `NUMBER_OF_PROCESSORS` variable from a +string to an integer. + +## Showing the Value of a Variable + +Let's show the value of the variable `HOME`: + +~~~ +$ echo HOME +~~~ +{: .language-bash} + +~~~ +HOME +~~~ +{: .output} + +That just prints "HOME", which isn't what we wanted +(though it is what we actually asked for). +Let's try this instead: + +~~~ +$ echo $HOME +~~~ +{: .language-bash} + +~~~ +/home/vlad +~~~ +{: .output} + +The dollar sign tells the shell that we want the *value* of the variable +rather than its name. +This works just like wildcards: +the shell does the replacement *before* running the program we've asked for. +Thanks to this expansion, what we actually run is `echo /home/vlad`, +which displays the right thing. + +## Creating and Changing Variables + +Creating a variable is easy — we just assign a value to a name using "=" +(we just have to remember that the syntax requires that there are _no_ spaces +around the `=`!): + +~~~ +$ SECRET_IDENTITY=Dracula +$ echo $SECRET_IDENTITY +~~~ +{: .language-bash} + +~~~ +Dracula +~~~ +{: .output} + +To change the value, just assign a new one: + +~~~ +$ SECRET_IDENTITY=Camilla +$ echo $SECRET_IDENTITY +~~~ +{: .language-bash} + +~~~ +Camilla +~~~ +{: .output} + +## Environment variables + +When we ran the `set` command we saw there were a lot of variables whose names +were in upper case. That's because, by convention, variables that are also +available to use by _other_ programs are given upper-case names. Such variables +are called _environment variables_ as they are shell variables that are defined +for the current shell and are inherited by any child shells or processes. + +To create an environment variable you need to `export` a shell variable. For +example, to make our `SECRET_IDENTITY` available to other programs that we call +from our shell we can do: + +~~~ +$ SECRET_IDENTITY=Camilla +$ export SECRET_IDENTITY +~~~ +{: .language-bash} + +You can also create and export the variable in a single step: + +~~~ +$ export SECRET_IDENTITY=Camilla +~~~ +{: .language-bash} + +> ## Using environment variables to change program behaviour +> +> Set a shell variable `TIME_STYLE` to have a value of `iso` and check this +> value using the `echo` command. +> +> Now, run the command `ls` with the option `-l` (which gives a long format). +> +> `export` the variable and rerun the `ls -l` command. Do you notice any +> difference? +> +> > ## Solution +> > +> > The `TIME_STYLE` variable is not _seen_ by `ls` until is exported, at which +> > point it is used by `ls` to decide what date format to use when presenting +> > the timestamp of files. +> > +> {: .solution} +{: .challenge} + +You can see the complete set of environment variables in your current shell +session with the command `env` (which returns a subset of what the command +`set` gave us). **The complete set of environment variables is called +your _runtime environment_ and can affect the behaviour of the programs you +run**. + +{% include {{ site.snippets }}/scheduler/print-sched-variables.snip %} + +To remove a variable or environment variable you can use the `unset` command, +for example: + +~~~ +$ unset SECRET_IDENTITY +~~~ +{: .language-bash} + +## The `PATH` Environment Variable + +Similarly, some environment variables (like `PATH`) store lists of values. +In this case, the convention is to use a colon ':' as a separator. +If a program wants the individual elements of such a list, +it's the program's responsibility to split the variable's string value into +pieces. + +Let's have a closer look at that `PATH` variable. +Its value defines the shell's search path for executables, +i.e., the list of directories that the shell looks in for runnable programs +when you type in a program name without specifying what directory it is in. + +For example, when we type a command like `analyze`, +the shell needs to decide whether to run `./analyze` or `/bin/analyze`. +The rule it uses is simple: +the shell checks each directory in the `PATH` variable in turn, +looking for a program with the requested name in that directory. +As soon as it finds a match, it stops searching and runs the program. + +To show how this works, +here are the components of `PATH` listed one per line: + +~~~ +/Users/vlad/bin +/usr/local/git/bin +/usr/bin +/bin +/usr/sbin +/sbin +/usr/local/bin +~~~ +{: .output} + +On our computer, +there are actually three programs called `analyze` +in three different directories: +`/bin/analyze`, +`/usr/local/bin/analyze`, +and `/users/vlad/analyze`. +Since the shell searches the directories in the order they're listed in `PATH`, +it finds `/bin/analyze` first and runs that. +Notice that it will *never* find the program `/users/vlad/analyze` +unless we type in the full path to the program, +since the directory `/users/vlad` isn't in `PATH`. + +This means that I can have executables in lots of different places as long as +I remember that I need to to update my `PATH` so that my shell can find them. + +What if I want to run two different versions of the same program? +Since they share the same name, if I add them both to my `PATH` the first one +found will always win. +In the next episode we'll learn how to use helper tools to help us manage our +runtime environment to make that possible without us needing to do a lot of +bookkeeping on what the value of `PATH` (and other important environment +variables) is or should be. + +{% include links.md %} diff --git a/_episodes/14-modules.md b/_episodes/15-modules.md similarity index 100% rename from _episodes/14-modules.md rename to _episodes/15-modules.md diff --git a/_episodes/15-transferring-files.md b/_episodes/16-transferring-files.md similarity index 100% rename from _episodes/15-transferring-files.md rename to _episodes/16-transferring-files.md diff --git a/_episodes/16-parallel.md b/_episodes/17-parallel.md similarity index 100% rename from _episodes/16-parallel.md rename to _episodes/17-parallel.md diff --git a/_episodes/17-resources.md b/_episodes/18-resources.md similarity index 100% rename from _episodes/17-resources.md rename to _episodes/18-resources.md diff --git a/_episodes/18-responsibility.md b/_episodes/19-responsibility.md similarity index 100% rename from _episodes/18-responsibility.md rename to _episodes/19-responsibility.md diff --git a/_includes/snippets_library/Birmingham_Baskerville_slurm/_config_options.yml b/_includes/snippets_library/Birmingham_Baskerville_slurm/_config_options.yml index 471f4fa0..8fbeb012 100644 --- a/_includes/snippets_library/Birmingham_Baskerville_slurm/_config_options.yml +++ b/_includes/snippets_library/Birmingham_Baskerville_slurm/_config_options.yml @@ -62,8 +62,9 @@ episode_order: - 11-connecting - 12-cluster - 13-scheduler - - 14-modules - - 15-transferring-files - - 16-parallel - - 17-resources - - 18-responsibility + - 14-environment-variables + - 15-modules + - 16-transferring-files + - 17-parallel + - 18-resources + - 19-responsibility diff --git a/_includes/snippets_library/ComputeCanada_Graham_slurm/_config_options.yml b/_includes/snippets_library/ComputeCanada_Graham_slurm/_config_options.yml index 2236091e..3dbf9cc9 100644 --- a/_includes/snippets_library/ComputeCanada_Graham_slurm/_config_options.yml +++ b/_includes/snippets_library/ComputeCanada_Graham_slurm/_config_options.yml @@ -62,8 +62,9 @@ episode_order: - 11-connecting - 12-cluster - 13-scheduler - - 14-modules - - 15-transferring-files - - 16-parallel - - 17-resources - - 18-responsibility + - 14-environment-variables + - 15-modules + - 16-transferring-files + - 17-parallel + - 18-resources + - 19-responsibility diff --git a/_includes/snippets_library/EPCC_Cirrus_slurm/_config_options.yml b/_includes/snippets_library/EPCC_Cirrus_slurm/_config_options.yml index 2a4587e3..57ab94fd 100644 --- a/_includes/snippets_library/EPCC_Cirrus_slurm/_config_options.yml +++ b/_includes/snippets_library/EPCC_Cirrus_slurm/_config_options.yml @@ -68,8 +68,9 @@ episode_order: - 11-connecting - 12-cluster - 13-scheduler - - 14-modules - - 15-transferring-files - - 16-parallel - - 17-resources - - 18-responsibility + - 14-environment-variables + - 15-modules + - 16-transferring-files + - 17-parallel + - 18-resources + - 19-responsibility diff --git a/_includes/snippets_library/HPCC_MagicCastle_slurm/_config_options.yml b/_includes/snippets_library/HPCC_MagicCastle_slurm/_config_options.yml index c57051fa..45f7a367 100644 --- a/_includes/snippets_library/HPCC_MagicCastle_slurm/_config_options.yml +++ b/_includes/snippets_library/HPCC_MagicCastle_slurm/_config_options.yml @@ -58,8 +58,9 @@ episode_order: - 11-connecting - 12-cluster - 13-scheduler - - 14-modules - - 15-transferring-files - - 16-parallel - - 17-resources - - 18-responsibility + - 14-environment-variables + - 15-modules + - 16-transferring-files + - 17-parallel + - 18-resources + - 19-responsibility diff --git a/_includes/snippets_library/Magic_Castle_EESSI_slurm/_config_options.yml b/_includes/snippets_library/Magic_Castle_EESSI_slurm/_config_options.yml index 724d0868..46d805a3 100644 --- a/_includes/snippets_library/Magic_Castle_EESSI_slurm/_config_options.yml +++ b/_includes/snippets_library/Magic_Castle_EESSI_slurm/_config_options.yml @@ -71,14 +71,15 @@ sched: comment: "#SBATCH" hist: "sacct -u yourUsername" hist_filter: "" - + episode_order: - 10-hpc-intro - 11-connecting - 12-cluster - 13-scheduler - - 14-modules - - 15-transferring-files - - 16-parallel - - 17-resources - - 18-responsibility + - 14-environment-variables + - 15-modules + - 16-transferring-files + - 17-parallel + - 18-resources + - 19-responsibility diff --git a/_includes/snippets_library/NIST_CTCMS_slurm/_config_options.yml b/_includes/snippets_library/NIST_CTCMS_slurm/_config_options.yml index 14948029..0533c725 100644 --- a/_includes/snippets_library/NIST_CTCMS_slurm/_config_options.yml +++ b/_includes/snippets_library/NIST_CTCMS_slurm/_config_options.yml @@ -63,7 +63,8 @@ episode_order: - 11-connecting - 12-cluster - 13-scheduler - - 15-transferring-files - - 16-parallel - - 17-resources - - 18-responsibility + - 14-environment-variables + - 16-transferring-files + - 17-parallel + - 18-resources + - 19-responsibility diff --git a/_includes/snippets_library/Norway_SIGMA2_SAGA_slurm/_config_options.yml b/_includes/snippets_library/Norway_SIGMA2_SAGA_slurm/_config_options.yml index 647ce293..80e2be9b 100644 --- a/_includes/snippets_library/Norway_SIGMA2_SAGA_slurm/_config_options.yml +++ b/_includes/snippets_library/Norway_SIGMA2_SAGA_slurm/_config_options.yml @@ -62,8 +62,9 @@ episode_order: - 11-connecting - 12-cluster - 13-scheduler - - 14-modules - - 15-transferring-files - - 16-parallel - - 17-resources - - 18-responsibility + - 14-environment-variables + - 15-modules + - 16-transferring-files + - 17-parallel + - 18-resources + - 19-responsibility diff --git a/_includes/snippets_library/UCL_Myriad_sge/_config_options.yml b/_includes/snippets_library/UCL_Myriad_sge/_config_options.yml index 149b3d97..79650711 100644 --- a/_includes/snippets_library/UCL_Myriad_sge/_config_options.yml +++ b/_includes/snippets_library/UCL_Myriad_sge/_config_options.yml @@ -41,8 +41,9 @@ episode_order: - 11-connecting - 12-cluster - 13-scheduler - - 14-modules - - 15-transferring-files - - 16-parallel - - 17-resources - - 18-responsibility + - 14-environment-variables + - 15-modules + - 16-transferring-files + - 17-parallel + - 18-resources + - 19-responsibility