Proposal for a notebook for a test case of environment variables list

[kartikmodi]

With a detailed input from @andre-merzky here and previous inputs by @mturilli on #1578 and some with discussion -

Idea: To make a notebook as a test case to ensure the pre-defined list of environment variables are in sync with usage, validity w.r.t. to the codebase.

Covering Cases:
If the environment variable is marked as deprecated in the list, it should not be used in code anymore, so documentation generation would fail.
If the environment variable is present in the list and not getting used anywhere in the codebase, then a warning will be thrown. This is because environment variables are known to be used in a dynamic fashion in code, which is out of scope, for now, so only warning no fail.

Plan:

1. Make a formatted predefined list e.g. in json/yaml file of env variable and its description, deprecation flag, etc. Manually with inputs from Documentation: Table with all the enviroment variables of rp  #1578
2. Read the list in the notebook and run some py codeparser which can look for specific syntaxes of code for accessing env variables, some of the syntaxes are -

os.environ['VAR']

os.environ.get('VAR')

os.getenv('VAR', default_value)

'VAR' in os.environ

for k,v in os.environ.items():
    if k.startswith('RP_'):
        self._task_env[k] = v

The single quotes could have been the double-quotes.

4. Test the above covering cases and take appropriate actions as per case.
5. The list will be embedded in an RST document, to be included in a published form.

Open doubts:
This seems to be a highly volatile task. Since it involves code parsing, less certainty, and could involve known scenarios of env variable usage.
The needs to explore -

1. Is this kind of work done on any other projects? How did they do? Search github repos. Search for pre-existing solutions.
2. There should be proper libraries to read code syntaxes as a language grammar, we should be able to read code as high-level tokens and not just as strings.

Future Scope -
Generate a list of variables from the codebase if possible. Don't know how? Static code parsing won't work.
Get a list of variables from a running RP session, production machine etc.

[kartikmodi]

The above is a BAD idea as what has been discussed.
After searching for properly extracting and documenting env variables, unfortunately, there's I didn't find an official (or wide accepted convention) way, to document those.

But someone was hinting a similar problem: edgi-govdata-archiving/web-monitoring-diff#27 but it's unanswered.

Based on @mtitov @andre-merzky ideas, here are certain ways we can document env variables -

1. [SOMEWHAT-REASONABLE-NOT_VERY_COMPLEX] - A custom role in sphinx like

 :env:`RADICAL_PILOT_DBURL`

We'll write this wherever this is being used, if used multiple times then might be multiple times, in docstring.
When the build will be generated it will refer to a URL in the documentation which is a #tag in the webpage.
The webpage can contain a list of all variables.

We need to write Role Processor in sphinx so, we might be able to get a list of USAGEs of env and we could dereference back from the list of the source.
Where to maintain the description, is_deprecated can be discussed, ie. in the source file where used or in global page.
Inspiration: https://doughellmann.com/posts/defining-custom-roles-in-sphinx/

2. [SOMEWHAT-MORE_PROMISING-MORECOMPLEX] - We use the sphinx domain API, which seems to be a more modern way of doing a similar thing to the above. It has cross-references with XRefRole. See this - https://opensource.com/article/18/11/building-custom-workflows-sphinx

3. [EASY-MIGHT_NOT_SOLVE_THE_PURPOSE] - Treating ENV variables as normal class variables, local variables -
   See https://www.sphinx-doc.org/en/master/usage/extensions/autodoc.html specially autodata
   Example - https://stackoverflow.com/questions/61314181/when-using-sphinx-how-can-i-document-members-that-dont-have-docstrings

    #: Doc comment for class attribute Foo.bar.
    #: It can have multiple lines.
    bar = 1

    flox = 1.5   #: Doc comment for Foo.flox. One line only.

    baz = 2
    """Docstring for class attribute Foo.baz."""

There is an open question of how this will behave with env variables since a lot of times they are accessed directly like os.environ.get('VAR') and not assigned to any variable before use.

Other URLs which actually creates confusion/ask questions - https://stackoverflow.com/questions/12527783/how-to-create-custom-directives-in-sphinx
Seems like we need to package it as a sphinx extension, as done in the first point link.

[andre-merzky]

A custom role in sphinx like :env:

There seems to be an :envvar:: directive in sphinx. I stumbled over that by chance. Would that be useful?

[kartikmodi]

Update -
The directive envvar works as expected but I found some limitations/observations behavior -

1. If the envvar RADICAL_TEST_ENV is used 2 times in the codebase, and is being referred (cross-linked) from somewhere else in code/docs, then the cross-linked reference points to the last declaration.

2. We need a global rst doc page where we could list all the env variables and put a reference to its usage.
   While I am able to do this when maintaining a global list manually or extracted by a regex but making it automated seems hard.
   Additionally, I am not able to figure out how to process envvar directive programmatically like the custom directive shown here - https://opensource.com/article/18/11/building-custom-workflows-sphinx

[kartikmodi]

Explanation with an example -

1. The test variable name is RADICAL_TEST_ENV. Url is generated in the form shown in the screenshot -
   

2.1 However, when we re-declare the same variable the URL is generated in the form - #envvar-0
2.2 Still unable to figure out how to write a description of envvar

3. Variables can be referred from anywhere with

:envvar:`RADICAL_TEST_ENV` dsaj sad

but if declared 2 times, then it will refer to the last declaration

@andre-merzky

[andre-merzky]

Given those problems and the effort spent on that topic, maybe we should reconsider the idea to manually compile a page with the relevant set of env variables?

As to the specific observations above, I don't really have any useful insight to provide, sorry :-/