From ab606f581fc22286a8f0cb83efd53469a59573a2 Mon Sep 17 00:00:00 2001 From: f-allian Date: Thu, 16 Nov 2023 15:46:40 +0000 Subject: [PATCH 1/2] Update: docs revamp --- README.md | 16 +- docs/README.md | 2 +- docs/source/conf.py | 18 ++- docs/source/credits.rst | 37 +++++ docs/source/description.rst | 42 ++--- docs/source/dev/actions_and_webhooks.rst | 24 +-- docs/source/dev/documentation.rst | 41 ++--- docs/source/dev/version_release.rst | 18 +-- docs/source/frontends/json_front_end.rst | 33 ++-- docs/source/frontends/test_suite.rst | 14 +- docs/source/glossary.md | 3 - docs/source/glossary.rst | 62 ++++++++ docs/source/index.rst | 153 +++++++++++++++++-- docs/source/installation.rst | 23 ++- docs/source/modules/causal_specification.rst | 29 +++- docs/source/modules/causal_tests.rst | 14 +- docs/source/modules/data_collector.rst | 36 +++-- docs/source/usage.rst | 32 ++-- 18 files changed, 422 insertions(+), 175 deletions(-) create mode 100644 docs/source/credits.rst delete mode 100644 docs/source/glossary.md create mode 100644 docs/source/glossary.rst diff --git a/README.md b/README.md index 8fc526b7..929179e5 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,15 @@ -# Causal Testing Framework: A Causal Inference-Driven Software Testing Framework +# Causal Testing Framework +### A Causal Inference-Driven Software Testing Framework -![example workflow](https://github.com/CITCOM-project/CausalTestingFramework/actions/workflows/ci-tests.yaml/badge.svg) [![codecov](https://codecov.io/gh/CITCOM-project/CausalTestingFramework/branch/main/graph/badge.svg?token=04ijFVrb4a)](https://codecov.io/gh/CITCOM-project/CausalTestingFramework) [![Documentation Status](https://readthedocs.org/projects/causal-testing-framework/badge/?version=latest)](https://causal-testing-framework.readthedocs.io/en/latest/?badge=latest) + +[![Project Status: Active – The project has reached a stable, usable state and is being actively developed.](https://www.repostatus.org/badges/latest/active.svg)](https://www.repostatus.org/#active) +![example workflow](https://github.com/CITCOM-project/CausalTestingFramework/actions/workflows/ci-tests.yaml/badge.svg) +[![codecov](https://codecov.io/gh/CITCOM-project/CausalTestingFramework/branch/main/graph/badge.svg?token=04ijFVrb4a)](https://codecov.io/gh/CITCOM-project/CausalTestingFramework) +[![Documentation Status](https://readthedocs.org/projects/causal-testing-framework/badge/?version=latest)](https://causal-testing-framework.readthedocs.io/en/latest/?badge=latest) +![Dynamic TOML Badge](https://img.shields.io/badge/dynamic/toml?url=https%3A%2F%2Fraw.githubusercontent.com%2FCITCOM-project%2FCausalTestingFramework%2Fmain%2Fpyproject.toml&query=%24.project%5B'requires-python'%5D&label=python) +![PyPI - Version](https://img.shields.io/pypi/v/causal-testing-framework) +[![DOI](https://t.ly/FCT1B)](https://orda.shef.ac.uk/articles/software/CITCOM_Software_Release/24427516) +![GitHub License](https://img.shields.io/github/license/CITCOM-project/CausalTestingFramework) Causal testing is a causal inference-driven framework for functional black-box testing. This framework utilises graphical causal inference (CI) techniques for the specification and functional testing of software from a black-box @@ -12,10 +21,9 @@ system-under-test that is expected to cause a change to some output(s). ![Causal Testing Workflow](images/workflow.png) - ## Installation -See the readthedocs site for [installation +See the Read the Docs site for [installation instructions](https://causal-testing-framework.readthedocs.io/en/latest/installation.html). ## Documentation diff --git a/docs/README.md b/docs/README.md index 5dadbaf7..de8abb3f 100644 --- a/docs/README.md +++ b/docs/README.md @@ -6,7 +6,7 @@ The documentation is accessible on [Read the Docs](https://causal-testing-framew ## Locally building To build locally, the requirements in `docs/source/requirements.txt` will need to be installed. - +[README.md](README.md) Within `docs/`, run `make html` to create or update the .html files in the `docs/build` directory. Running `make clean` will clean the `build` folder. \ No newline at end of file diff --git a/docs/source/conf.py b/docs/source/conf.py index a9aa633a..5c680d97 100644 --- a/docs/source/conf.py +++ b/docs/source/conf.py @@ -12,6 +12,7 @@ # import os import sys +from datetime import date # Include root directory of project sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), "../.."))) @@ -19,9 +20,9 @@ # -- Project information ----------------------------------------------------- -project = "Causal Testing" -copyright = "2021, Andrew Clark, Michael Foster, Neil Walkinshaw, Rob Hierons, Bob Turner" -author = "Andrew Clark, Michael Foster, Neil Walkinshaw, Rob Hierons, Bob Turner" +project = "Causal Testing Framework" +copyright = f"{date.today().year}, The CITCOM Team" +author = "Andrew Clark, Michael Foster, Neil Walkinshaw, Rob Hierons, Bob Turner, Christopher Wild, Farhad Allian" # -- General configuration --------------------------------------------------- @@ -49,7 +50,16 @@ # Add any paths that contain custom static files (such as style sheets) here, # relative to this directory. They are copied after the builtin static files, # so a file named "default.css" will overwrite the builtin "default.css". -# html_static_path = ['_static'] +html_static_path = ['_static'] + +html_css_files = ['css/custom.css'] # Path to generate documentation from using sphinx AutoAPI autoapi_dirs = [os.path.abspath(os.path.join("..", "..", "causal_testing"))] + +html_logo = '_static/images/CITCOM-logo.png' + +html_theme_options = { + 'style_nav_header_background': '#9ADBE8', # Set the colour using CSS + 'logo_only' : True, +} \ No newline at end of file diff --git a/docs/source/credits.rst b/docs/source/credits.rst new file mode 100644 index 00000000..6a82e48e --- /dev/null +++ b/docs/source/credits.rst @@ -0,0 +1,37 @@ +Acknowledging CITCOM +********************* + +If you use the Causal Testing Framework for your work, please add the following line within your paper to acknowledge +our work: + +.. admonition:: Important + :class: important + + This research has used version X.Y.Z (software citation) of the Causal Testing Framework (paper citation). + +The paper citation should be the Causal Testing Framework paper_, and the software citation should be the specific +Figshare DOI_ for the version used in your research. + +.. _paper: https://dl.acm.org/doi/10.1145/3607184 +.. _DOI: https://dl.acm.org/doi/10.1145/3607184 + +.. code-block:: latex + + @ARTICLE{Clark_etal_2023, + author = {Clark, Andrew G. and Foster, Michael and Prifling, Benedikt and Walkinshaw, Neil and Hierons, Robert M. + and Schmidt, Volker and Turner, Robert D.}, + title = {Testing Causality in Scientific Modelling Software}, + year = {2023}, + publisher = {Association for Computing Machinery}, + url = {https://doi.org/10.1145/3607184}, + doi = {10.1145/3607184}, + journal = {ACM Trans. Softw. Eng. Methodol.}, + month = {jul}, + keywords = {Software Testing, Causal Testing, Causal Inference} + } + +The CITCOM Team +*************** + +- Andrew G. Clark, Michael Foster, Richard Somers, Christopher Wild, Farhad Allian, Neil Walkinshaw. + diff --git a/docs/source/description.rst b/docs/source/description.rst index 514e62ce..c15f78f6 100644 --- a/docs/source/description.rst +++ b/docs/source/description.rst @@ -1,29 +1,28 @@ -Framework Overview +Background ===================================== -CTF Components --------------- +The Causal Testing Framework consists of 3 main components: 1) Causal Specification, 2) Causal Test Case and 3) Data Collection. #. - :doc:`Causal Specification <../modules/causal_specification>`\ : Before we can test software, we need to obtain an - understanding of how it should behave in a particular use-case scenario. In addition, to apply graphical CI + :doc:`Causal Specification <../modules/causal_specification>`\ : To apply graphical CI techniques for testing, we need a causal DAG which depicts causal relationships amongst inputs and outputs. To collect this information, users must create a *causal specification*. This comprises a set of scenarios which place constraints over input variables that capture the use-case of interest, a causal DAG corresponding to this scenario, and a series of high-level functional requirements that the user wishes to test. In causal testing, these requirements should describe how the model should respond to interventions (changes made to the input configuration). + + #. :doc:`Causal Tests <../modules/causal_tests>`\ : With a causal specification in hand, we can now go about designing a series of test cases that interrogate the causal relationships of interest in the scenario-under-test. Informally, - a causal test case is a triple (M, X, Delta, Y), where M is the modelling scenario, X is an input configuration, - Delta is an intervention which should be applied to X, and Y is the expected *causal effect* of that intervention on - some output of interest. Therefore, a causal test case states the expected causal effect (Y) of a particular - intervention (Delta) made to an input configuration (X). For each scenario, the user should create a suite of causal + a causal test case is a triple ``(M, X, Delta, Y)``, where ``M`` is the modelling scenario, ``X`` is an input configuration, + ``Delta`` is an intervention which should be applied to ``X``, and ``Y`` is the expected *causal effect* of that intervention on + some output of interest. Therefore, a causal test case states the expected causal effect (``Y``) of a particular + intervention (``Delta``) made to an input configuration (``X``). For each scenario, the user should create a suite of causal tests. Once a causal test case has been defined, it is executed as follows: - - #. Using the causal DAG, identify an estimand for the effect of the intervention on the output of interest. That is, + a. Using the causal DAG, identify an estimand for the effect of the intervention on the output of interest. That is, a statistical procedure capable of estimating the causal effect of the intervention on the output. #. Collect the data to which the statistical procedure will be applied (see Data collection below). #. Apply a statistical model (e.g. linear regression or causal forest) to the data to obtain a point estimate for @@ -35,6 +34,8 @@ CTF Components test should pass or fail based on the results. In the simplest case, this takes the form of an assertion which compares the point estimate to the expected causal effect specified in the causal test case. + + #. :doc:`Data Collection <../modules/data_collector>`\ : Data for the system-under-test can be collected in two ways: experimentally or observationally. The former involves executing the system-under-test under controlled @@ -48,22 +49,3 @@ CTF Components provide instructions for an execution that will fill the gap (future work). For more information on each of these steps, follow the link to their respective documentation. - -Causal Inference Terminology ----------------------------- - -Here are some explanations for the causal inference terminology used above. - - -* Causal inference (CI) is a family of statistical techniques designed to quantify and establish **causal** - relationships in data. In contrast to purely statistical techniques that are driven by associations in data, CI - incorporates knowledge about the data-generating mechanisms behind relationships in data to derive causal conclusions. -* One of the key advantages of CI is that it is possible to answer causal questions using **observational data**. That - is, data which has been passively observed rather than collected from an experiment and, therefore, may contain all - kinds of bias. In a testing context, we would like to leverage this advantage to test causal relationships in software - without having to run costly experiments. -* There are many forms of CI techniques with slightly different aims, but in this framework we focus on graphical CI - techniques that use directed acyclic graphs to obtain causal estimates. These approaches used a causal DAG to explain - the causal relationships that exist in data and, based on the structure of this graph, design statistical experiments - capable of estimating the causal effect of a particular intervention or action, such as taking a drug or changing the - value of an input variable. diff --git a/docs/source/dev/actions_and_webhooks.rst b/docs/source/dev/actions_and_webhooks.rst index aeaa38b3..030745e8 100644 --- a/docs/source/dev/actions_and_webhooks.rst +++ b/docs/source/dev/actions_and_webhooks.rst @@ -1,26 +1,30 @@ Github Actions and Webhooks =========================== -Github Actions +Actions -------------- -Currently this project makes use of 3 `Github Actions `_, which can be found in the `.github/workflows `_ directory +Currently, this project makes use of 3 `Github Actions `_, +which can be found in the +`.github/workflows `_ directory. They are: -#. ci-tests.yaml, which runs CI tests on each PR +#. ``ci-tests.yaml``, which runs continuous integration (CI) tests on each on each pull request. -#. lint-format.yaml, runs linting on each PR +#. ``lint-format.yaml``, which runs linting on each pull request. -#. publish-to-pypi.yaml, runs when a new version tag is pushed and publishes that tag version to pypi +#. ``publish-to-pypi.yaml``, runs when a new version tag is pushed and publishes that tag version to PyPI. -Github Webhooks +Webhooks --------------- -The project also uses 3 `Webhooks `_, which can be found in the `project settings `_ on Github +The project also uses 3 `Webhooks `_, which can +be found in the `project settings `_ on Github. These +include: -#. To codacy +#. `Codacy `_ -#. To codecov +#. `Codecov `_ -#. To readthedocs \ No newline at end of file +#. `Read the Docs `_ \ No newline at end of file diff --git a/docs/source/dev/documentation.rst b/docs/source/dev/documentation.rst index a561a560..71b7284c 100644 --- a/docs/source/dev/documentation.rst +++ b/docs/source/dev/documentation.rst @@ -1,27 +1,23 @@ Project Documentation ===================== -This page aims to describe: +This page describes: -#. The projects documentation style +#. The projects documentation style, -#. The tools used for documentation +#. The tools used for documentation, -#. ReadTheDocs where the documentation for this project is hosted +#. Read the Docs where the documentation for this project is hosted. -Documentation Style +Style and Tools ------------------- -The `Sphinx docstring format `_ is used throughout the projects codebase to allow for the easy understanding of classes, methods, functions, etc. -This format allows for the easy generation of html documentation pages. +The `Sphinx docstring format `_ is used +throughout the projects codebase to allow for the easy understanding of classes, methods, functions, etc. This format +allows for the easy generation of html documentation pages. It also checks for docstrings that have been added to the projects PyLint configuration. -Checks for docstrings have been added to the projects Pylint configuration. - -Documentation Tools -------------------- - -To install the packages required to work with the documentation please ensure the projects **dev** dependencies are installed:: +To install the packages required to work with the documentation, please ensure the projects **dev** dependencies are installed:: pip install causal-testing-framework[dev] @@ -42,20 +38,25 @@ To cleanup the compiled docs you can run:: make clean -Situation within `docs/source` are the reStructuredText files (.rst) which contain the handwritten doc pages, which get compiled by the make commands. -Autodoc & Autoapi +The source reStructuredText files (.rst) files can be found within `docs/source` directory, which contains the +handwritten documentation pages and get compiled by the ``make`` commands. + +Autodoc & AutoAPI ***************** -`Autodoc `_ is an extension to sphinx that can import code modules and compile documentation from their docstrings. +`Autodoc `_ is an extension to Sphinx that can import +code modules and compile documentation from their docstrings. -`AutoAPI `_ is a third party sphinx tool for recursively discovering code modules and compiling them into a logical doctree structure +`AutoAPI `_ is a third party Sphinx tool for recursively discovering +code modules and compiling them into a logical doctree structure -The configuration for Sphinx, Autodoc and AutoAPI are all found in `/docs/source/conf.py `_. +The configuration for Sphinx, Autodoc and AutoAPI can all be found in `/docs/source/conf.py `_. ReadTheDocs ----------- -`Read the Docs `_ is a documentation hosting site that hosts, versions and builds documentation for free for open source projects. +`Read the Docs `_ is a documentation hosting site that hosts, versions and builds documentation +for free for open source projects. -This project makes use of a Github Webook to trigger the build in ReadTheDocs, further reading on this can be found :doc:`here <../dev/actions_and_webhooks>`\ +This project makes use of a Github Webhook to trigger the build in ReadTheDocs, further reading on this can be found :doc:`here <../dev/actions_and_webhooks>`\ diff --git a/docs/source/dev/version_release.rst b/docs/source/dev/version_release.rst index 6dde2c2d..85264bb8 100644 --- a/docs/source/dev/version_release.rst +++ b/docs/source/dev/version_release.rst @@ -2,28 +2,28 @@ Version Releases ================ This project follows the `Semantic Versioning 2.0.0 `_ style for code releases. -This page aims to describe the release process for a new code version on the `Project Github `_ +This page describes the release process for a new framework version on the `Project Github `_. How to release -------------- -#. Once your PR(s) are merged, navigate to https://github.com/CITCOM-project/CausalTestingFramework/releases, which can be found on the right hand side of the projects Github main page by clicking on 'Releases' +#. Once your PR(s) are merged, navigate to https://github.com/CITCOM-project/CausalTestingFramework/releases, which can be found on the right hand side of the projects Github main page by clicking on 'Releases'. -#. Press the **Draft a new release** button in the top right of the releases page +#. Press the **Draft a new release** button in the top right of the releases page. -#. Press the **Choose a tag** button and add the new version following the Semantic Version guidelines. Please include the 'v' before the tag, e.g. **v0.0.0** +#. Press the **Choose a tag** button and add the new version following the Semantic Version guidelines. Please include the 'v' before the tag, e.g. **v0.0.0**. -#. Enter the same tag name into the **Release Title** box +#. Enter the same tag name into the **Release Title** box. -#. Press **Generate Release Notes** button +#. Press **Generate Release Notes** button. #. Add any additional information that may be helpful to the release notes. If there are breaking changes for example, which modules will they affect? -#. Ensure the **Set as the latest release** checkbox is selected +#. Ensure the **Set as the latest release** checkbox is selected. -#. Press publish release +#. Press publish release. -#. Check that the Github Action worker, found in the `Actions tab `_ on the github, has successfully completed. Typical time to publish to PyPI is around 2 minutes. +#. Check that the Github Action worker, found in the `Actions tab `_ has successfully completed. The typical time to publish to PyPI is around 2 minutes. #. Check on the projects `PyPI page `_ that the latest release is ready! diff --git a/docs/source/frontends/json_front_end.rst b/docs/source/frontends/json_front_end.rst index dcff7928..a18a389f 100644 --- a/docs/source/frontends/json_front_end.rst +++ b/docs/source/frontends/json_front_end.rst @@ -1,23 +1,22 @@ JSON Frontend ====================================== -The JSON frontend allows Causal Tests and parameters to be specified in JSON to allow for tests to be quickly written -whilst retaining the flexibility of the Causal Testing Framework (CTF). +The JSON frontend allows causal tests and parameters to be specified in JSON to allow for tests to be quickly written +whilst retaining the flexibility of the framework. -basic workflow +Basic Workflow -------------- The basic workflow of using the JSON frontend is as follows: -#. Specify your test cases in the JSON format (more details below) -#. Create your DAG in a dot file -#. Initialise the JsonUtility class in python with a path of where you want the outputs saved -#. Set the paths pointing the Json class to your json file, dag file and optionally your data file (see data section below) using the :func:`causal_testing.json_front.json_class.JsonUtility.set_paths` method -#. Run the :func:`causal_testing.json_front.json_class.JsonUtility.setup` method providing your scenario +#. Specify your test cases in the JSON format (more details below). +#. Create your DAG in a dot file. +#. Initialise the JsonUtility class in python with a path of where you want the outputs saved. +#. Set the paths pointing the Json class to your json file, dag file and optionally your data file (see data section below) using the :func:`causal_testing.json_front.json_class.JsonUtility.set_paths` method. +#. Run the :func:`causal_testing.json_front.json_class.JsonUtility.setup` method providing your scenario. #. Run the :func:`causal_testing.json_front.json_class.JsonUtility.run_json_tests` method, which will execute the test cases provided by the JSON file. Example Walkthrough ------------------- -An example is provided in `examples/poisson` which will be walked through in this README to better understand -the framework +An example is provided in `examples/poisson` which contains a README with more detailed information. run_causal_tests.py ******************* @@ -31,11 +30,11 @@ such as: #. Meta constraint functions #. Mapping JSON distributions, effects, and estimators to python objects -Use case specific information is also declared here such as the paths to the relevant files needed for the tests. +Use-case specific information is also declared here such as the paths to the relevant files needed for the tests. causal_tests.json ***************** -`examples/poisson/causal_tests.json `_ contains python code written by the user to implement scenario specific features +`examples/poisson/causal_tests.json `_ contains Python code written by the user to implement scenario specific features is the JSON file that allows for the easy specification of multiple causal tests. Tests can be specified two ways; firstly by specifying a mutation lke in the example tests with the following structure: Each test requires: @@ -60,7 +59,7 @@ The second method of specifying a test is to specify the test in a concrete form Run Commands ************ -This example uses the Argparse utility built into the JSON frontend, which allows the frontend to be run from a commandline interface as shown here. +This example uses the ``Argparse`` utility built into the JSON frontend, which allows the frontend to be run from a commandline interface as shown here. To run the JSON frontend example from the root directory of the project, use:: @@ -79,13 +78,9 @@ The behaviour of where the log file is produced and named can be altered with th Runtime Data ----------- +------------- There are currently 2 methods to inputting your runtime data into the JSON frontend: #. Providing one or more file paths to `.csv` files containing your data -#. Setting a dataframe to the .data attribute of the JsonUtility instance, this must be done before the setup method is called. - - - - +#. Setting a dataframe to the .data attribute of the JSONUtility instance, this must be done before the setup method is called. \ No newline at end of file diff --git a/docs/source/frontends/test_suite.rst b/docs/source/frontends/test_suite.rst index 94d10d92..8ddd0afa 100644 --- a/docs/source/frontends/test_suite.rst +++ b/docs/source/frontends/test_suite.rst @@ -4,12 +4,12 @@ Test Suite The test_suite feature allows for the effective running of multiple causal_test_cases using a logical structure. This structure is defined by the parameters in the class: :class:`causal_testing.testing.causal_test_suite`. -A current limitation of the Test Suite is that it requires references to the estimator class not instances (objects) of -estimator classes. Which prevents the usage of some of the features of an estimator class. +A current limitation of the Test Suite is that it requires references to the estimator class, not instances (objects) of +estimator classes, which prevents the usage of some of the features of an estimator. -Test Suite - Class +Class -------------------- -The test_suite class is an extension of the python UserDict_, meaning it simulates a standard python dictionary where +The test_suite class is an extension of the python UserDict_, meaning it simulates a standard Python dictionary where any dictionary method can be used. The class also features a setter to make adding new test cases quicker and less error prone :meth:`causal_testing.testing.causal_test_suite.CausalTestSuite.add_test_object`. @@ -20,12 +20,12 @@ the value is a test object in the format of another dictionary: test_object = {"tests": causal_test_case_list, "estimators": estimators_classes, "estimate_type": estimate_type} -Each `base_test_case` contains the treatment and outcome variables and only causal_test_cases testing this relationship -should be placed in the test object for that `base_test_case`. +Each ``base_test_case`` contains the treatment and outcome variables, and only causal_test_cases testing this relationship +should be placed in the test object for that ``base_test_case``. .. _UserDict: https://docs.python.org/3/library/collections.html#collections.UserDict -Test Suite - Execution +Execution ----------------------- The test_suite can be executed by a call to the :meth:`causal_testing.testing.causal_test_engine.CausalTestEngine.execute_test_suite`. Here the causal_test_engine will iterate over all the test objects and execute each `test` once per `estimator` and per diff --git a/docs/source/glossary.md b/docs/source/glossary.md deleted file mode 100644 index b0160c1c..00000000 --- a/docs/source/glossary.md +++ /dev/null @@ -1,3 +0,0 @@ -# Glossary - -TDB diff --git a/docs/source/glossary.rst b/docs/source/glossary.rst new file mode 100644 index 00000000..4e57012b --- /dev/null +++ b/docs/source/glossary.rst @@ -0,0 +1,62 @@ +.. Glossary + +Glossary +################### + +.. glossary:: + + Causal inference + Causal inference (:abbr:`CI`) is a family of statistical techniques designed to quantify and establish **causal** + relationships in data. In contrast to purely statistical techniques that are driven by associations in data, CI + incorporates knowledge about the data-generating mechanisms behind relationships in data to derive causal conclusions. + + Causal DAG + A Directed Acyclic Graph depicting the direct causal relationships between variables, in which an edge ``X -> Y`` indicates that ``X`` directly causes ``Y``. That is, there exists an intervention on ``X`` which brings about a change in ``Y``. + + Treatment Variable + The changed variable of interest (``X``). + + Outcome Variable + The observed variable of interest (``Y``). + + Causal Specification + A causal specification is a pair ``S = (M, G)`` comprising a modelling scenario ``M`` and a causal DAG ``G`` capturing the causal relationships amongst the inputs and outputs of the SUT that are central to the modelling scenario. + + Causal Test Case + A causal test case is a 4-tuple ``(M, X, |delta|, Y)`` that captures the expected causal effect, Y, of an intervention, ``delta``, made to an input valuation, ``X``, on some model outcome in the context of modelling scenario ``M``. + + Directed acyclic graph (DAG) + A directed acyclic graph (:abbr:`DAG`) is a graphical representation used in causal inference to model and visualize relationships between variables. + In a DAG, nodes represent variables, and directed edges between nodes indicate causal relationships, with the absence of cycles ensuring acyclicity. + + Estimate Type + The effect measure to use, typically (C)ATE, Risk Ratio, or Odds Ratio + + ATE + ~~~~ + **Average treatment effect** (:abbr:`ATE`): The additive difference in the outcome between the control and treatment populations. + + CATE + ~~~~~~~~~~~ + **Conditional ATE** (:abbr:`CATE`): The additive difference in the outcome between the control and treatment populations across different strata of the population. + + Risk Ratio + ~~~~~~~~~~~ + **Risk ratio**: The multiplicative difference in the outcome between the control and treatment populations. + + Odds Ratio + ~~~~~~~~~~~ + **Odds Ratio**: The ratio of the odds of A in the presence of B and the odds of A in the absence of B. + + + Minimal Adjustment Set + The smallest set of variables which must be controlled, or "adjusted for", to produce an unbiased estimate of causal effect. + + Scenario + A modelling scenario ``M`` is a pair ``(X, C)`` where ``X`` is a non-strict subset of the model's input variables and ``C`` is a set of constraints over valuations of ``C``, which may be empty. + + Scenario Execution + A software execution satisfying a given modelling scenario. + + Intervention + An intervention ``delta : X -> X'`` is a function which manipulates the values of a subset of input valuations. \ No newline at end of file diff --git a/docs/source/index.rst b/docs/source/index.rst index 3e41b28f..c083e96f 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -1,19 +1,101 @@ -.. Causal Testing documentation master file, created by - -Welcome to Causal Testing's documentation! +Welcome to the Causal Testing Framework ========================================== -Causal testing is a causal inference-driven framework for functional black-box testing. This framework utilises +|status| |ci-tests| |code-cov| |docs| |python| |pypi| |doi| |license| + +Overview +******* + +Causal testing is a :term:`causal inference`-driven framework for functional black-box testing. This framework utilises graphical causal inference (CI) techniques for the specification and functional testing of software from a black-box -perspective. In this framework, we use causal directed acyclic graphs (DAGs) to express the anticipated cause-effect +perspective. In this framework, we use causal :term:`directed acyclic graphs` (DAGs) to express the anticipated cause-effect relationships amongst the inputs and outputs of the system-under-test and the supporting mathematical framework to design statistical procedures capable of making causal inferences. Each causal test case focuses on the causal effect of an intervention made to the system-under test. That is, a prescribed change to the input configuration of the system-under-test that is expected to cause a change to some output(s). -.. image:: /images/workflow.png +.. raw:: html + + + +.. raw:: html + + + + + +.. container:: zoom-container + + .. image:: /images/workflow.png + :class: zoomable-image + :alt: Zoomable Image + .. toctree:: + :hidden: + :caption: Home + + self + +.. toctree:: + :hidden: :maxdepth: 1 :caption: Introduction @@ -22,6 +104,7 @@ system-under-test that is expected to cause a change to some output(s). usage .. toctree:: + :hidden: :maxdepth: 1 :caption: Module Descriptions @@ -32,11 +115,13 @@ system-under-test that is expected to cause a change to some output(s). .. toctree:: :maxdepth: 2 :caption: API + :hidden: :titlesonly: /autoapi/causal_testing/index .. toctree:: + :hidden: :maxdepth: 1 :caption: Front Ends @@ -44,21 +129,67 @@ system-under-test that is expected to cause a change to some output(s). frontends/test_suite .. toctree:: + :hidden: :maxdepth: 1 :caption: Glossary glossary .. toctree:: + :hidden: :maxdepth: 1 - :caption: Development Documentation + :caption: Development /dev/version_release /dev/documentation /dev/actions_and_webhooks -Indices and tables -================== +.. toctree:: + :caption: Useful Links + :hidden: + :maxdepth: 2 + + Paper + Figshare + PyPI + +.. toctree:: + :hidden: + :maxdepth: 1 + :caption: Credits + + credits + +.. Define variables for our GH badges + +.. |ci-tests| image:: https://github.com/CITCOM-project/CausalTestingFramework/actions/workflows/ci-tests.yaml/badge.svg + :target: https://github.com/CITCOM-project/CausalTestingFramework/actions/workflows/ci-tests.yaml + :alt: Continuous Integration Tests + +.. |code-cov| image:: https://codecov.io/gh/CITCOM-project/CausalTestingFramework/branch/main/graph/badge.svg?token=04ijFVrb4a + :target: https://codecov.io/gh/CITCOM-project/CausalTestingFramework + :alt: Code coverage + +.. |docs| image:: https://readthedocs.org/projects/causal-testing-framework/badge/?version=latest + :target: https://causal-testing-framework.readthedocs.io/en/latest/?badge=latest + :alt: Documentation + +.. |python| image:: https://img.shields.io/badge/dynamic/toml?url=https%3A%2F%2Fraw.githubusercontent.com%2FCITCOM-project%2FCausalTestingFramework%2Fmain%2Fpyproject.toml&query=%24.project%5B'requires-python'%5D&label=python + :target: https://img.shields.io/badge/dynamic/toml?url=https%3A%2F%2Fraw.githubusercontent.com%2FCITCOM-project%2FCausalTestingFramework%2Fmain%2Fpyproject.toml&query=%24.project%5B'requires-python'%5D&label=python + :alt: Python + +.. |pypi| image:: https://img.shields.io/pypi/v/causal-testing-framework + :target: https://pypi.org/project/causal-testing-framework/ + :alt: PyPI + +.. |status| image:: https://www.repostatus.org/badges/latest/active.svg + :target: https://www.repostatus.org/#active + :alt: Status + +.. |doi| image:: https://t.ly/FCT1B + :target: https://orda.shef.ac.uk/articles/software/CITCOM_Software_Release/24427516 + :alt: DOI -* :ref:`genindex` -* :ref:`modindex` \ No newline at end of file +.. |license| image:: https://img.shields.io/github/license/CITCOM-project/CausalTestingFramework + :target: https://github.com/CITCOM-project/CausalTestingFramework + :alt: License \ No newline at end of file diff --git a/docs/source/installation.rst b/docs/source/installation.rst index 86f42c1b..7e6c07a8 100644 --- a/docs/source/installation.rst +++ b/docs/source/installation.rst @@ -1,23 +1,32 @@ -Installation +Getting started ============ Requirements ------------ -CausalTestingFramework requires python version 3.9 or later -If installing on Windows, ensure `Microsoft Visual C++ `_ is version 14.0 or greater +* Python >= 3.9. +* `Microsoft Visual C++ `_ 14.0+ (Windows only). + -Pip Install +Installation ----------- +The Causal Testing Framework can be installed through either the `Python Package Index (PyPI)`_ (recommended), or directly from source. + +.. _Python Package Index (PyPI): https://dl.acm.org/doi/10.1145/3607184 + +Method 1: Installing via pip +.............................. + To install the Causal Testing Framework using :code:`pip` for the latest stable version:: pip install causal-testing-framework -To install with development packages/tools:: +or if you want to install with development packages/tools:: pip install causal-testing-framework[dev] -Install from source -------------------- + +Method 2: Installing via source +............................... To install from source:: diff --git a/docs/source/modules/causal_specification.rst b/docs/source/modules/causal_specification.rst index f8c24412..083a5a5b 100644 --- a/docs/source/modules/causal_specification.rst +++ b/docs/source/modules/causal_specification.rst @@ -2,27 +2,42 @@ Causal Specification ==================== -In causal testing, our units of interest are specific usage **scenarios** of the system-under-test. For example, when testing an epidemiological computational model, one scenario could focus on the simulation of the spread of a virus in a crowded indoors space. For this scenario, our causal specification will describe how a number of interventions should **cause** some outputs to change e.g. opening a window should reduce the spread of the virus by some factor. +- In causal testing, our units of interest are specific usage **scenarios** of the system-under-test. For example, + when testing an epidemiological computational model, one scenario could focus on the simulation of the spread of a virus in a crowded indoors space. + For this scenario, our causal specification will describe how a number of interventions should **cause** some outputs to change e.g. opening a window should reduce the spread of the virus by some factor. -In order to isolate the causal effect of the defined interventions, the user needs to express the anticipated cause-effect relationships amongst the inputs and outputs involved in the scenario. This is achieved using a causal DAG, a simple dot and arrow graph that does not contain any cycles where nodes are random variables that represent the inputs and outputs in the scenario-under-test, and edges represent causality. For example, ``window --> infection_prob`` encodes the belief that opening or closing the window should cause the probability of infection to change. +- In order to isolate the causal effect of the defined interventions, the user needs to express the anticipated cause-effect relationships amongst the inputs and outputs involved in the scenario. + This is achieved using a causal DAG, a simple dot and arrow graph that does not contain any cycles where nodes are random variables that represent the + inputs and outputs in the scenario-under-test, and edges represent causality. For example, ``window --> infection_prob`` encodes the belief that opening or closing the + window should cause the probability of infection to change. -A causal specification is simply the combination of these components: a series of requirements for the scenario-under-test and a causal DAG representing causality amongst the inputs and outputs. We will discuss these in more detail below. +- A causal specification is simply the combination of these components: a series of requirements for the scenario-under-test and a causal DAG representing causality + amongst the inputs and outputs. We will discuss these in more detail below. Scenario Requirements --------------------- -Each scenario is defined as a series of constraints placed over a set of input variables. A constraint is simply a mapping from an input variable to a specific value or distribution that characterises the scenario in question. For example, a scenario simulating the spread of a virus in a crowded indoors space would likely place a constraint over the size of room, the number of windows, and the number of people in the room. +- Each scenario is defined as a series of constraints placed over a set of input variables. A constraint is simply a mapping + from an input variable to a specific value or distribution that characterises the scenario in question. + For example, a scenario simulating the spread of a virus in a crowded indoors space would likely place a constraint over the size of room, + the number of windows, and the number of people in the room. -Requirements for this scenario should describe how a particular intervention (e.g. opening the window, changing the number of people, changing the size of the room etc.) is expected to cause a particular outcome (number of infections, deaths, R0 etc.) to change. The way these requirements are expressed is up to the user, however, it is essential that they focus on the expected effect of an intervention. +- Requirements for this scenario should describe how a particular intervention + (e.g. opening the window, changing the number of people, changing the size of the room etc.) is expected to cause a particular outcome (number of infections, deaths, R0 etc.) to change. + The way these requirements are expressed is up to the user, however, it is essential that they focus on the expected effect of an intervention. Causal DAG ---------- -In order to apply CI techniques, we need to capture causality amongst the inputs and outputs in the scenario-under-test. Therefore, for each scenario, the user must define a causal DAG. While there is generally no guidance/algorithm that can be followed to create a causal DAG, there are a couple of requirements that should be satisfied. +In order to apply CI techniques, we need to capture causality amongst the inputs and outputs in the scenario-under-test. +Therefore, for each scenario, the user must define a causal DAG. While there is generally no guidance/algorithm that can be followed to create a causal DAG, +there are a couple of requirements that should be satisfied. #. The DAG must contain all inputs and outputs involved in the scenario-under-test. #. If there are any other variables which are not directly involved but are expected to have a causal relationship with the variables in the scenario-under-test, these should also be added to the graph. For example, the size of the room might be partially caused by the simulated location (house styles, average wealth etc.), in which case location should be added to the DAG with an edge to room size and any other variables it is deemed to influence. #. If in doubt, add an edge. It is a stronger assumption to exclude an edge (X and Y are independent) than to include one (X has some potentially negligiable causal effect on Y). -Collectively, the components of the causal specification provide both contextual information in the form of constraints and requirements, as well as causal information in the form of a causal DAG. Later on, these components will be used to design statistical experiments that can answer causal questions about the scenrio-under-test, such as ``Does opening a window impair the viruses ability to spread?`` +- Collectively, the components of the causal specification provide both contextual information in the form of constraints and requirements, + as well as causal information in the form of a causal DAG. Later on, these components will be used to design statistical experiments that + can answer causal questions about the scenario-under-test, such as ``Does opening a window impair the viruses ability to spread?`` diff --git a/docs/source/modules/causal_tests.rst b/docs/source/modules/causal_tests.rst index 36ef3227..e8207c19 100644 --- a/docs/source/modules/causal_tests.rst +++ b/docs/source/modules/causal_tests.rst @@ -4,9 +4,9 @@ Causal Testing This package contains the main components of the causal testing framework, causal tests and causal oracles, which utilise both the specification and data collection packages. -A causal test case is a triple ``(X, \Delta, Y)`` where ``X`` is an input configuration, ``\Delta`` is an intervention, and ``Y`` is the expected causal effect of applying ``\Delta`` to ``X``. Put simply, a causal test case states the expected change in an outcome that applying an intervention to X should cause. In this context, an intervention is simply a function which manipulates the input configuration of the scenario-under-test in a way that is expected to cause a change to some outcome. +- A causal test case is a triple ``(X, \Delta, Y)`` where ``X`` is an input configuration, ``\Delta`` is an intervention, and ``Y`` is the expected causal effect of applying ``\Delta`` to ``X``. Put simply, a causal test case states the expected change in an outcome that applying an intervention to X should cause. In this context, an intervention is simply a function which manipulates the input configuration of the scenario-under-test in a way that is expected to cause a change to some outcome. -For example, suppose we have an epidemiological computational model and we are testing the model in a classroom scenario. In particular, we are interested in how various precautions, such as hand washing and mask wearing, can prevent the spread of the virus in a classroom. Let us walk through the steps of causal testing. +- For example, suppose we have an epidemiological computational model and we are testing the model in a classroom scenario. In particular, we are interested in how various precautions, such as hand washing and mask wearing, can prevent the spread of the virus in a classroom. Let us walk through the steps of causal testing. Specification ------------- @@ -45,18 +45,18 @@ We then define a number of causal test cases to apply to the scenario-under-test Data Collection --------------- -To run these test cases experimentally, we need to execute both ``X`` and ``\Delta(X)`` - that is, with and without the interventions. Since the only difference between these test cases is the intervention, we can conclude that the observed difference in ``n_infected_t5`` was caused by the interventions. While this is the simplest approach, it can be extremely inefficient at scale, particularly when dealing with complex software such as computational models. +- To run these test cases experimentally, we need to execute both ``X`` and ``\Delta(X)`` - that is, with and without the interventions. Since the only difference between these test cases is the intervention, we can conclude that the observed difference in ``n_infected_t5`` was caused by the interventions. While this is the simplest approach, it can be extremely inefficient at scale, particularly when dealing with complex software such as computational models. -To run these test cases observationally, we need to collect *valid* observational data for the scenario-under-test. This means we can only use executions with between 20 and 30 people, a square environment of size betwen 20x20 and 40x40, and where a single person was initially infected. In addition, this data must contain executions both with and without the intervention. Next, we need to identify any sources of bias in this data and determine a procedure to counteract them. This is achieved automatically using graphical causal inference techniques that identify a set of variables that can be adjusted to obtain a causal estimate. Finally, for any categorical biasing variables, we need to make sure we have executions corresponding to each category otherwise we have a positivity violation (i.e. missing data). In the worst case, this at least guides the user to an area of the system-under-test that should be executed. +- To run these test cases observationally, we need to collect *valid* observational data for the scenario-under-test. This means we can only use executions with between 20 and 30 people, a square environment of size betwen 20x20 and 40x40, and where a single person was initially infected. In addition, this data must contain executions both with and without the intervention. Next, we need to identify any sources of bias in this data and determine a procedure to counteract them. This is achieved automatically using graphical causal inference techniques that identify a set of variables that can be adjusted to obtain a causal estimate. Finally, for any categorical biasing variables, we need to make sure we have executions corresponding to each category otherwise we have a positivity violation (i.e. missing data). In the worst case, this at least guides the user to an area of the system-under-test that should be executed. Causal Inference ---------------- -After collecting either observational or experimental data, we now need to apply causal inference. First, as described above, we use our causal graph to identify a set of adjustment variables which mitigate all bias in the data. Next, we use statistical models to adjust for these variables (implementing the statistical procedure necessary to isolate the causal effect) and obtain the desired causal estimate. Depending on the statistical model used, we can also generate 95% confidence intervals (or confidence intervals at any confidence level for that matter). +- After collecting either observational or experimental data, we now need to apply causal inference. First, as described above, we use our causal graph to identify a set of adjustment variables which mitigate all bias in the data. Next, we use statistical models to adjust for these variables (implementing the statistical procedure necessary to isolate the causal effect) and obtain the desired causal estimate. Depending on the statistical model used, we can also generate 95% confidence intervals (or confidence intervals at any confidence level for that matter). -In our example, the causal DAG tell us it is necessary to adjust for ``environment`` in order to obtain the causal effect of ``precaution`` on ``n_infected_t5``. Supposing the relationship is linear, we can employ a linear regression model of the form ``n_infected_t5 ~ p0*precaution + p1*environment`` to carry out this adjustment. If we use experimental data, only a single environment is used by design and therefore the adjustment has no impact. However, if we use observational data, the environment may vary and therefore this adjustment will look at the causal effect within different environments and then provide a weighted average, which turns out to be the partial coefficient ``p0``. +- In our example, the causal DAG tell us it is necessary to adjust for ``environment`` in order to obtain the causal effect of ``precaution`` on ``n_infected_t5``. Supposing the relationship is linear, we can employ a linear regression model of the form ``n_infected_t5 ~ p0*precaution + p1*environment`` to carry out this adjustment. If we use experimental data, only a single environment is used by design and therefore the adjustment has no impact. However, if we use observational data, the environment may vary and therefore this adjustment will look at the causal effect within different environments and then provide a weighted average, which turns out to be the partial coefficient ``p0``. Test Oracle Procedure --------------------- -After conducting causal inference, all that remains is to ascertain whether the causal effect is expected or not. In our example, this is simply a case of checking whether the causal effect on ``n_infected_t5`` falls within the specified range. However, in the future, we may wish to implement more complex oracles. +- After conducting causal inference, all that remains is to ascertain whether the causal effect is expected or not. In our example, this is simply a case of checking whether the causal effect on ``n_infected_t5`` falls within the specified range. However, in the future, we may wish to implement more complex oracles. diff --git a/docs/source/modules/data_collector.rst b/docs/source/modules/data_collector.rst index 508f897e..be1e73c7 100644 --- a/docs/source/modules/data_collector.rst +++ b/docs/source/modules/data_collector.rst @@ -1,24 +1,28 @@ Data Collection =============== -For causal testing, we require data for the scenario-under-test. This data can be collected in two ways: experimentally +For causal testing, we require data for the scenario-under-test. This data can be collected in 2 ways: experimentally and observationally. -Experimental data collection involves running the system-under-test with two specific input configurations, one with the -intervention and one without. We refer to these as the treatment and control configurations, respectively. The only -difference between these two input configurations is the intervention and therefore the observed difference in outcome -is the causal effect. If the system-under-test is non-deterministic, each input configuration should be ran multiple -times to observe the difference in the distributions of outputs. +Experimental Data Collector +**************************** +- Experimental data collection involves running the system-under-test with two specific input configurations, one with the + intervention and one without. We refer to these as the treatment and control configurations, respectively. The only + difference between these two input configurations is the intervention and therefore the observed difference in outcome + is the causal effect. If the system-under-test is non-deterministic, each input configuration should be ran multiple + times to observe the difference in the distributions of outputs. -Observational data collection involves collecting past execution data for the system-under-test that was not ran under -the experimental conditions necessary to isolate the causal effect. Instead, we will use the causal knowledge encoded -in the causal specification's causal DAG to identify and appropriately mitigate any sources of bias in the data. That -way, we can still obtain the causal effect of the intervention but avoid running costly experiments. +Observational Data Collector +***************************** -We cannot use any data as observational data, though. We need to ensure that the data is representative of the -scenario-under-test. To achieve this, we filter any provided data using the defined constraints by checking whether the -data for a variables falls within the specified distribution or meets the exact specified value. +- Observational data collection involves collecting past execution data for the system-under-test that was not ran under + the experimental conditions necessary to isolate the causal effect. Instead, we will use the causal knowledge encoded + in the causal specification's causal DAG to identify and appropriately mitigate any sources of bias in the data. That + way, we can still obtain the causal effect of the intervention but avoid running costly experiments. -This package should contain methods which collect the data for causal inference. Users must implement these methods in a -way that generates (experimental) or collects (observational) data for the scenario-under-test. For the observational -case, we should also provide helper methods which filter the data. \ No newline at end of file +- We cannot use any data as observational data, though. We need to ensure that the data is representative of the + scenario-under-test. To achieve this, we filter any provided data using the defined constraints by checking whether the + data for a variables falls within the specified distribution or meets the exact specified value. + +- This package should contain methods which collect the data for causal inference. Users must implement these methods in a way that generates (experimental) or collects + (observational) data for the scenario-under-test. For the observational case, we should also provide helper methods which filter the data. \ No newline at end of file diff --git a/docs/source/usage.rst b/docs/source/usage.rst index 7341280e..842fe3f1 100644 --- a/docs/source/usage.rst +++ b/docs/source/usage.rst @@ -2,14 +2,13 @@ Usage ----- -There are currently three methods of using the Causal Testing Framework, through -the :doc:`JSON Front End `\, using -:doc:`Test Suites `\, or directly as +There are currently 3 methods of using the Causal Testing Framework; 1) :doc:`JSON Front End `\, 2) +:doc:`Test Suites `\, or 3) directly as described below. -The causal testing framework is made up of three main components: Specification, Testing, and Data Collection. The first +The causal testing framework is made up of 3 main components: Specification, Testing, and Data Collection. The first step is to specify the (part of the) system under test as a modelling ``Scenario``. Modelling scenarios specify the -observable variables and any constraints which exist between them. We currently support three types of variable: +observable variables and any constraints which exist between them. We currently support 3 types of variable: * ``Input`` variables are input parameters to the system. @@ -77,19 +76,12 @@ data, e.g. .. code-block:: python - data_csv_path = 'results/data.csv' - data_collector = ObservationalDataCollector(modelling_scenario, data_csv_path) + obs_df = pd.read_csv('results/data.csv') + data_collector = ObservationalDataCollector(modelling_scenario, obs_df) -The actual running of the tests is done using the ``CausalTestEngine`` class. The setup of the test engine is as follows: - -.. code-block:: python - - from causal_testing.testing.causal_test_engine import CausalTestEngine - - causal_test_engine = CausalTestEngine(causal_specification, data_collector) # Instantiate the causal test engine Whether using fresh or pre-existing data, a key aspect of causal inference is estimation. To actually execute a test, we -need an estimator. We currently support two estimators: linear regression and causal forest. The estimators require the +need an estimator. We currently support two estimators: linear regression and logistic regression. The estimators require the minimal adjustment set from the causal_dag. This and the estimator can be instantiated as per the `documentation `_. @@ -98,18 +90,18 @@ the `documentation `_ -feature +feature. From e997aac63cd6842c6778d9e9f725ba2da46ed7d3 Mon Sep 17 00:00:00 2001 From: f-allian Date: Wed, 6 Dec 2023 11:17:21 +0000 Subject: [PATCH 2/2] Update: corrected docs and uploaded logo --- docs/source/_static/css/custom.css | 29 +++++++++++++++++++ docs/source/_static/images/CITCOM-logo.png | Bin 0 -> 20244 bytes docs/source/conf.py | 10 ++++++- docs/source/frontends/json_front_end.rst | 17 +++++++---- docs/source/index.rst | 11 +++---- docs/source/installation.rst | 6 ++-- docs/source/modules/causal_specification.rst | 2 +- docs/source/modules/causal_tests.rst | 3 -- docs/source/usage.rst | 3 +- 9 files changed, 60 insertions(+), 21 deletions(-) create mode 100644 docs/source/_static/css/custom.css create mode 100644 docs/source/_static/images/CITCOM-logo.png diff --git a/docs/source/_static/css/custom.css b/docs/source/_static/css/custom.css new file mode 100644 index 00000000..70b8a18d --- /dev/null +++ b/docs/source/_static/css/custom.css @@ -0,0 +1,29 @@ +/* Custom CSS */ + +/* Note: /* +The CSS below will only work if 'logo_only':True in conf.py, but will require adjusting if False. + */ + +.wy-side-nav-search a .logo { + max-width: 200px; + max-height: 170px; + height: auto; + width:auto; +} + + + +.wy-nav-top a { + color: #131E29; /* Change the colour of the homepage navigation title on mobiles */ +} + +.wy-menu-vertical p.caption { + + color: #9ADBE8; /* Change the colour of captions*/ +} + + +.rst-content code.literal, .highlight .s2, .highlight .s1, .rst-content tt.literal{ + color: #440099; + font-weight: bold; +} \ No newline at end of file diff --git a/docs/source/_static/images/CITCOM-logo.png b/docs/source/_static/images/CITCOM-logo.png new file mode 100644 index 0000000000000000000000000000000000000000..af9f058ab34adb9ca54911a8d6dd8cbbefa78874 GIT binary patch literal 20244 zcmX_o1yodB+xCE7oX}f6Ho4dFhIhldn-Q8I}+gUrC7&(}+*gIKd919YHK;$4fNwN1HnTN|3 zhB|-d-<)aaS^fqW`p%fgee4LjZoogOEXvFGgn7_q(Dv*+hGl!Hh>N9#4fe;vs8bNAp1u& zG4Cs_CTg1e-{Vd+FzBgM5y3Gd{43(+5t-;ow(#ZQ@18mKU;D1+Sp?=;cXK(R3?V9( zcyf{m>0tlCU~rf&c1(2-hU_D?a&(r*Yj89KfkK?)V8rsN1*;;b(dfIkY}`%KtsF)x z2U0j5HMx}|uOwzX`H6yVYyv)oB*ub}kL6=Ae3r;%m-)|TW;C-PGR*NviNDEjRr?s0 zao;$i$r}e@eETcH>rXX4E85z)x}2LS%7jDZYs=*1n+W5Yu&}%y_co7O8o@CCQ zK|P;e7Yr&?DG~7=`o!id$?#^>(IaiBH0YbTMKk0x-hNrpVxxP|_!KmO@vA zeD;=p!zacVsN*Q~4Yu+S?a%%Ymki{Hm4@DhSt6PLu-R5*S4ba3jvhzxoT*KnjTc&- zr-~>2W^Tmo^6*ei+l*%+x!sf^KQs$ z+Y!z9gCrt0i6i7`^NV=cY_Ugaga&M+9!_*}4$IXo+-neJIDXn}fzkGWLT-#!&2@d6 z4SmN2`8_%s0f#Mp6i1M_7xr$MZ+L4zz_sRi8AoRvqP9GPR_JJL|@g2GjGJ%Yszqh;Mg(ov7?XW zPtpQ1YLAP7nICvCXLA9`H!|g11U`6Gji1V>FZ+ih^K6Auk_82ijbj!w+&@!*gZ)DJ zT5_h*5A?38TBQba7OE~r@1JAkQAj4)8zKvgnZDE@bT^B^O4DQ>E^@>_GhXjcQn60w zbq3*nX+nmI$&W8Lm$}bJ(*17n2&os6&!0Q(831;L>EVpd*)=Xy-yEwKY5k#FQpKbS z%Iq<3*qrFyE>B*owHp)N@I32*-=+mlaFLT1-4zAu7tYVhFJ4h)G%W>XP?rHXh;a8^G zr{Z9U`f7_0T%mQG--}|uTgH5hMC?y{3G%UBomFx1Fp1=G&0Y>GBb#wQa=ho7XISYi zK6O?V@oDuPTe$Yty1nHoIzve)m$yxK#>^@p22UIp z^eMb#P&%-?+Ht zV()8l#`g339e&MSrZ<%Ahy=C=2byRp39Mm$KGZJ8nu~pV_HMsZ(YSKjzBd1eac4Y{ z7ab})A*CU?WkkSfTiBJlEZYL#GfN;T)IZFyz70qG3cX8320y`~_(dR?i%J!^K;|#b zP!zJ>rPAw}mc)NM+SojF`~GCTL!HvJ()ewF>>X42AiNQY3C@CDkh5-n(%oO$q*4-kMbYwjCWLZF4Rez+ zlV;xfMjg|c1`wCqYW5@E{)c`#+r-bN2)_z7!*#1^9&#Bz%}M7&7~PBpkkvc2Ci-sYmzxG{kvbhaosZD)u3}**loTU^J%Xd#WSbd9JSsSg zyHZo(l@idMo161~fd~CyDDjW?5Fr@eHDhP<8#Da^rA;rBbPDwjD;U@Von|d{jar$1 z{4MUkheyWW?J#Hn#|?C)?msvAstDz&cj~fLMw=KDEX2iYO@hhnxy=SQWIRog$VUP3 z%t)Q4cdy$#)HbuQ>(->B8oL!~)mzZAU zMI_0FU9`` zh|dX36-LP^t%owxfpY`Gzy*kIN zx1Z*0=*uQRElG$mbbJTBi)<1mBpau9PXERn%f|fEnqwOf+@PeOb2iXb#^SIMu0pp1 z3j$pA*BaY$cNAu#MYyQ`i$85Wf=C*G$~M#Mc3FDkL2Uwvyqq4}s5b4}qE@>$@-TkZ zH43QR>;5~lxqf|`rw_j*W=lKLS79^2wteI&;B1I?i_P6J+nEfls!lti_xX`oqX&@b zyl*7`(}pf|{{xb&v62%}qW#zDvwy$aUmyLwcD3gbUxhppnCd1Xg_AMwVITod9%QEk z$2sfJnzWAE$PD#v`;YSAdEzfb75kQ?)GPxLLmgvc5OUZ*k3oN3ZYjxXvhrBIJQbQg zO9&S=$T`Qvec0LxXHIpI-Q2diD_R(jh>rvHcYIRaIZhNQoU5b^*%e}b```V+%?5WG zfAHyvjBU_|ki~!BIl@Xyf&fCs$rUF~4>s`!*DTxB8?Bz8-H&*hkm$6izR(WuX)yI1 z%AD_?D)2x(OWJ$DxdP!0TLY26K=nmPd-;@gQQ&g|%wcO0wp+@8|Hs)P3J=w2dV>$-TXW?}+*aF8>MC}OEZ-U5R6H1wSgerYCEx|6b!JfjUd{u$APSZf zmhGS;t=y;1>p{(X7iLb8`)eL%zic3>PSsL$xEPiRS(qW{RPG*C6DtdcEtHL}nc3+dAD;+lOAs&D&5(Ey zPDu6jB<_vLgQi!TZ;i}&jJ{2$6}n>Y2o)P>u$^wv$ce5JYKJ%CASwYslo3cPHM(H` z-5Wt?##RD|{J&Be9JhP(iKk)x6f3zLnKvc^h}7=Jam4bzufAjl z05&(Maq^hmrqdkVnqny*qR!4ysU&u7n-+Jk;`^meVDIl0z2_d=JvxfR-yElpl;$EF z$drf`l9$v_4rp+&TlT~loODlsNs6A&$@w^{l%(#f5-2oVt2(|{>c{>?b+uev8uD!6 zJScS8TFF8MZ?ncikIP{;2B0K+GuGQ3niH z$7Rlf)^=; zFoq<#)x#DvjxJ*u^7?sFUjG}Qt!#A!?ApQzwr8L(?$+#u+np9l)UXLGilW^)&O(0) zNYjV`nF^Z%6blHr!hU_TG9uX&X2^(|Hb7E}K>EvzvnA#v`$v87y?WTTv3Asl)M#^U znF{<9G9aG;_g|~cRZKDZ6v(NWT<6zfu(R5Hnx*K0Dx@R7$pF1P*;^SPn{f~A2?r#K zfTxJ+AjD;Q(hL+pm9x zs9ueKaygm!B8h_z=)&|cX1u2lA$Bg3d#FSqG=h<2Gif)meERunG^asq_0S`~P9t0j zfe}G$p$yaI49t<&^!n#%O~uHtM1Bf^hZ;`hXx0 zW~eu)a(rw-6=0U5g4(GwO3E|G&m*rG{4oBB#u@3rHk-T=+<)KE#Xy$u-kcFbSFda~ zEswZ;p_dkg>baD1@t_zLJ1q2utd#u6F;c)AD4jg8(v_*{piDX=9EgQj^2dXqtMwti zJ%nC_;o$!k2-}6XXGt5wxnLtxKqOE}NyQC@K!+(W(M_~2J0>r!ceOyaIo)G^&zCV2 zXe!&ukK=2E)NMj@ED>sqDK4lyt0FY#bKt8_C-6$tu;sOdPi+9&z$FD&mMp>|`k8!t-WSRv}lmrnN&B#=g z*oW3?HifovI8N>cD&bwOshje_J{l)%KR=U%&b!}4Q$6*SK}vpfe8a8#M# ze-4)#Y!lX-^AD5xM{AG$S!I1Jc3&kf*r|7RVPPIfP(sJjY0AgBZkAA0mJ@L2KDR&QNDERx`#>%f=K!R z8N#1oXOvFA0;D~QJ5f}mjw7#aWH)-gIch^={s2U zKdOBX64be?yBz^&ebhls(VypnR@$E?j`u^|?8#7=U(XfY)qY~aqTr=N03-K+o5_iD zgDK+Z=}WRIErIibrvYvF#N+ziRYM@8R2r0FZ7&Aozmz+n6laefkjjX{Ah&c!FU;CS zn>~B`K0^zEkk;1kQ`d$~MaU*p{EGOf)v2S^QL(*T>}Vy5uWPhn(|(hwHxvJjQ-uqX z5yc`5S)22aq%NRjVkf|Si^>*mje6sf-&v2?+rErHc`ZJ>`07biq(A!09MgwX>1ehL zMGz=W7p+7b2KdudMc^`8M?i=a6_uo>)c6g~&V+4V-Phw7va8QO7$D{HTM5ZN`UDur zMhFG8p0dRILpQ-6a7>L5(M*L?YIF9q3EdX5BN?>(_EK2AcbchegqdzWHIfn45NhgC z21#`>a02paJ-Vl?!533wmmM6Hkzd!P#=)jwmkU4V&64t|8`mX(;2y%rECbdA1EtHnr5 zpjE`=jE>IpJ+XUl`vP<2<=)H1{r7L9pfoiu6*uD>zF{q+{-4PR5fP4eUcT4g-l(3v zYW$HHml1oQ|0S_LR(Ubff|YNPAU7+;_ijr;HluZ-WVY@jvn9iPYTN{0rbllF_w$F)kVOX@7eR(QEnthae= zy*L)mX3M#46)|f&FD}R3-SHhCO4qkG3S7zLDkNyDNH90FUolv@+lMaiWH;N>k|b~M z6*CRE6J_@uy5bo>(`faG#O?I^EX(HL3aogdn(1_em>KshuV;SmY{yycxa9Kne##lt zbLCFa^<)k(*;pc(c?-Cc_reB|EsYt5jmk%>?UEGIh9AfQ?L?5B6wN9R;1EYFkw*f; z45xZ*w^owEe1+p^Tzml;_TQ%dxkDhmK+=0@p%a`KH-U+k8WN#JFLogClx-s?rJPK9 zMzZ5H;`(m&fSy|arJ_~BNb`PFx=4S#M9xN0B*6B?7yb>mFG|Lrzat6)S&kkE0xLeR zyhJ?7oLSpYA}gC-%PaY}_EN&uDk>VDSdba+w%QvSPuCW;k8EzzKiwk}FFewiGwFTR z7Ax=Ic!S zR8f0JkuQ-C6SaDgH7o36HHs){)i3A;d$$c(&N-Q*pM+Mil6`9@^AqO)YAU_C;lD~( zmNNz@K$9WdhhEhY3T2?AFWSI@>a<_7wvsYs2+MtE*1US}-3FCb(I7S5ALv z!hW)4Mo^G-a;Hz^uk*~Gdc!-_p%Z&>b-UN?b$Z3=chLgp*5=qwY9MmK)8>2dXRor# z(VZ6(Z1VxR!NvpkWYs-%mZp01BQx%y_leTu#otKPPWNx8Cn83@&tB+8FYTecbk$uh zs5>G8LNQhfC9vb6)j)*Xg~W4zp;`nB$&EhGFfeC6fNP>GIb`#nxYj|I%@8 z??V=Z1Wtz}ZhFpFQM1O#Z!{{7K5c)g%Axw`V#5m_#5cY^!fT2`xluu=8u=UHrpA7SzeGTP!C0eN{~ z3^2T=DjPNf4mAgXDH07+A)>Y-7CsL?o85#-aoMZDtRwtGo6v?5m=BS=%MNp?uI7hL zMy#*V`1Flv)c`;3!*9Nn08=x<`9U5CnoXBeW0HmwRL|byqzhX)HevkuB0-`kVLsx^ zi-+OWzZw{B^qma#Au_(kFuYZQubYC7b&uCdUheSJeRXY^jx^1!0Z)NiHd8aKyw8H% z@dFHHf(Av3;uHIFlUVFERp{ggwUCw#y*=CVj1S(~;*`nc5Y)2Ca$HWA1^uz7KNPuk zC4$}Co~kltd@B#VNBFZUe!m3xsgUP~KTtS+pfKhz7cJ$zs+O)~!ou1s77l2Lkg()$ zLa*2k6}%6f^H}9=c(Ov5CM5QYnH;86y_FSyUz7bYT9TB$_pozB{IR))_qK=w|I<8P z0hjLtO6^LNrwWcZ_1U512f%>D${Pdj`+cire~f!SZ__@f_r>9A;T1YFFD=>yYX>Z+ zR%HmopeMT0(=_3aaf7JKyU3dJkj07BnE)ygJF&Z)4EDMCCKV7 zb+>|+-eB_!pZjv?VF9Ba{^WFBQu5l9$P&yH?~SGv-32;IPv!+{U->)htouOmQdRP| ze%uM?7AE_evw+PHr4sQ8jUamttW0MpYLDf76bx<}H-g2Ig8(x5im%bk186>hfY@Yx zg!Nng)>n=?pQ)jR#-jr?*+-hxi&^ilc>>8JqR4`uNn5p?fz z^V(it-Df+x&18tx&0*KhZg#2@EV7DxH^J<3KzZLVMGCLJw1A!wqMWwxk&?FC6)NDO zQTxa`LmMJEzIC#m5ijgWd*61Q6lQi!Q{(~{MAP$nJ=IC;>V6jQtyxQ8gub9$--un6 z7_GFmBg>ixiXzQNAfV9u@!_pFj`rz@jU*Ea9zDlyT*;5r7hQ1AZs__%{&sa*+Ady( zh+Qzlhj5_q3?Fss_w!qYQM}@w=S5$<644AvcEzONn>o>t6^o39QN(Y%Muv zET2L`d|Hi&&8x!H@2~K*J?1qrnmuRTER<-9WCUtIR7v^iAfFsKf1>Ld;H> zjP)Yy8(-1bF;vjR%g4LVG^0JB)-@`(lIC+d=uN0OT;IRdTQveA-=o{}aGApQMttsn z=9_$b7fCfV*XZMAgO)TcHhE5bWAd)>ngxhAU`5(%VD#!Azex>@X9j1idXR;C2D5}& zeE|e~Gem2puQcNro?c7`}p5y(HsJnX^P-6MW<8GRu> zH%VQ_4v*yn^X65X*X0YEjIhld(XqK31I%a-G+R4Vf^W^_PuAJON#<@5Px$0P^txU! zEZ%FZq8ZRhFT8+8*BerdXFe}& zamt0F$+^6XSnvH=U})8jT*^@IJlCcQX(s=^1v`=P>Mi8jrb>6n3f7C>K;A^!K)OsS$8FGB`Vl#T&aetq3!9s_F zi503qde=%mysHOz!}UiI;Q&uOz2jrmL2Ufc*EPGn?w;+^m6rp+Zcs()cXOutOrQI- z44+QOTo$elKM?&QH8E%xI6&t^7z1BJT-@b5X>n@G@(32`Uel`H1ysM&YK}4FFJ+jR z{Y@<@Nyh-hc*+IKO;PmoY_v5e`o7s(wtRqHKz_NL`|Ox0_deb0+AnT@{&~#0Emni? zV5Su#KJ#y0SkUIkiOrM1D1Ie@kxi^VX?R^(;WfsJPPk;(5*^YSS+C%^jYsU0von0< zc!E5S-~NWIvT5xn)#oSk(5EKhMH9}?nAR-wu?^=EZ7f_~N5NskLm3~~f)taDs>y)x zZ}TcWXvup--|K56<+(D@UQcG7;b>qC*oJnzs|47769A3Z8K6>IMzXE)9tj_Oqv;dc z`4pkihhkAO-%b(&Gw@D6tA^kEH?`Z^u!^UiBo20zCeO0)ks(4_>F=t}HBDJ9rg+1n zljEE8J{j@P9#x3uLKmYfH^(+JX1hhLgMHvI9g^O;m0BIf+ZA?}hcao`-eTqz3Q;bR z&7a>TD3d8?5s?#>@9=E!4`l#z-p0c5lEB!0p9Xto$6c0u(|d4e#t4p2*vSiPOLQHH zxOY#@$$}e4k^Du|qg8y$b%#D;s^@0#@F~!L@=y^iWrhaW7@E$645U(ejkr-MvADJ@?Sz>ss-}zXm%=j6C`eW4kuL*e6ev4!aiiXq~P;)di`b@ze@V*%D zYb5hqz!ZI|E?Bj3*$+oxE_lP6wW8}VcBi};;VO=~*~-eqFBF;0h|y>z>=LJy{G~Yu zGc7WunT_&fx+T1MLlQ8alsjH_Z&}=jnpjXVJBREcxPb)Ai+Uoi?n!`uEB|9na!vI2 zSn^<9{aau6_I#ZzwfHmPK#db{`!dFrp@Y8pVJ$vp4DrG|JM_*IoBkdeeT`4@^xTQ4 zN2_k^4MlqUI$k|Z=g1o5*NQu+6BQZ6v9g&ZYfBe=6%DSM@Uf18q<>n8(Ss6@cPZz> z-wcyJ5a}@E)@h|&d~qyI+~Gn@D^2+s*mf^qP5SKeyS*LHe0gj5FO58+u@>iQv5(%y zmz@kppd34MgfxarOcDwC7^7lPyzu<0|LSpk&Ux!J#ojP2)nRV@BQk#se+uLa(O(bS zkP8iKjAZ3x=CB8CMcT`Hu8S{9PV2;YQk(5x+^*Og`YIzr6t|z9K|-3-2@bcxO1>M7 zqoxw8zRxn+N^icw8L_8%u!d}GV1$;E%PWC?rev;9-w35ZTK?WU`{hu9e9!GO`+CM zZUJ}snzBM9A=Hs%{g__@2gI`5W6Ml_OyRBGDqwOK4xyez(u` zwvwX;Q+eAJ1AXb9kZ%LvK;N_UB<8 z8^PH$F5MeyvAx^Nv=k*;eH@dNOnJ!tyeSA*Ti*bSB7xDG*N8iHX2*?Xh_}PP086v8?99xis86i<$ zQ2fKF7&WFCiFhP-j78&up-ql=5_wAGI@zcxN=#)W3-+2LR{zrh{aE)XA=U^pCO6jU=x6Pa`t) z_aSY)|7*GvpW2N|#qMvSX4pz%wyIn~Up;EIez_2r%B!4FT_leJ+6iaZXU+R51X&&t zB$H))1USG^$Up-69+5vDiEj3T3}`tgJ_jk!{Rx+$!NGQ9IB~JOkk*e)i~-CgR*2ZZ z!y3}8J}4eJ7WTy+bJJZSZV*3)oLrLJTT4^Uw8M*#9ji9pGruWnKNZaJHXZ@!K?(nE z#1ASn_Y(vxZGbaz&b{F8A}Ng_;g#9lb3rAhc;ghT$S37{QqT)gfa3b?R|<+sQ|_~e zG5%KhuKg~KUzv~}45j_O-xP&BF>7^ftJ8Gd9}zPGYG<8G%OUPxL;p)4Nv;(#8Yi}` z=^-GGKZ!bN_XvQPPG}-{ocbqStCo6xAz{ zS*6o*r=U!;s$0)<=wS@PP`G>g`}5P%g&zWijNFe$yPqF80Snt-SNsYji8Ahvz*!S; zhmU;_WG`zEgp;2bL~~2Qq46Y?6;NvPVDFq5M@H@S+~(P06)lX6!bkyPznQ*{=Hd zrUr6V>!u-ZjEl955#}Yg{g{9c+=J%IqNC=g5Gz$;?N4uWOdeqqAd3eIW(~c*?MkS7 z<&`dTu?H8mkF_MM434kaK`VcFfLas_VhBd%4?#bVG`c;2@eLatNswDkB092md2Hjb zf*2KDw}SZtmaV#xWwn5GHbT|*3%bx*xe-QDARL`NEtEnzUA-||`)Co5VX@hIVr?VI zQ_#*<#MvHiVuSAq4GQ`CjEmtg(S7%2D3gA z3RX7%Q8g|V??DGZC^wTfEN@bEw5C0Y8pMC4Q|~0a7ytFq5f}TJA@nJ=5WDrjYC_Lo z^Yp5@Yz()qG!g&S6Fu5bm2DSq{>UxL@=goTloo3vEXzSvcMZ#^J%H@4VP zeA`e0xTc=O+pI4NJa8L!V7a$#_ttK{WHGI4LllSN%9AR^%D2RWP{)lFks~_M1uI{Jv4s(DU(`2UB5uLz5E1HYc**D-i$Nx8o}~UX~^EAZo%xz$i1L z#_(+wp7UD~@i0a+CheXwzoSaq%$T@S8usBdEl;Z(bi=xg)uEG)A5V*R+5>b|aQO%+ zRMZBy`|nuPcSm;XD19CTs>DRrZs#q#`F5T88%J=IWsTD!iP70(XFq)iSSMZZl31d1 zJ%-hPqfZzbI^p4OjuC^GKnCjyd*RDY7|m||l+c{+^~lF=SCuv>3B>0!e_hcY9QoKj zQsW|jICSTr^X^gGPn!sGtO_s$N}Xqco9(ZHnj)4|!nb<(+Vq`0A~*NJ29y8-^p>2warIEUlDpiNYi)x0nq2 z4q)GOJ_gCFf*XJ7A2%U&pOv~l!s)SUX7cI(zB@B4735jq9OK_O)l4S(+f}zBhEpa< zKYR_Zf&s*3KLYYT8w}_{gD@IYyexP2D*gz5TUInbg-)CBhpkF&)B;+6XTP$(03ovW z&7QEfSbnwOiTM#^K`VobH#S^^0`jXiqPyr!gw=Q}FLTDtPLd0wvN`5s$^UFklWG zXV#{q*~bnuW7yS>+a#!@PFhG(um#?Qe0VmcQKWJi(|+lOsq>o}n#%^DBwJ!jctp*K z?^$Gy(=@7n#`|%1lEqfC?RVY-g)?ycNL9*4jJv^rD&CVi)iD^8C>BVKJlhMM4sMDW zSN4Gf3BbLWE)NP45}o9T{!yq0fSsO)9|Yt{OANhnX#w=N@4L?Q>|<^qPt^FLXY&D# zQiC&-7}YP4mc2$Y0B80kWHuNmgWhhOiR9pVIasC-2~@N_^R$(Yfx39c1<^JqzDH7G z#?*GyB3he(yVR>2p0msWxGdjxdz%`N*5sBMJwFubrhs1X3;e-K2Bud0CkxH~UJ*~q zA&$>9)@M%Cr{y6NK<)0O?R2Xy&u$htSkK2txDSQmE4)8Q`|?ms8Rz z-$I@GCmHYaQy=B#6j@5?XNl&hBeyctzD^-fWcN)WPH4=Y>X{;rNvoi0NTz@V0)G4j zcD5Hx)w6mqP+iU3&a=EVdYb<-Q{lcox>y-F%6}JgIYbPUb0`z0*Z_tVl4QmKN1nUA z33c2WV9u2Pi7{Sxsm1gCG4#cOpL?Sis7l)S9-nA?-8{8@*lEdFwi~!^)bG1)G?04# zrEsc}sGc6S#zKm-vV*gZQ0;yXJn|&-b!ug5>Z3IC+)Zm>`NOu;$xsO8oIrMP7L)rQ z!s)}MU(Ll{;`+jm(}Z|3eW=!$k;7wsJ;Rkfl{c#!u9KtVLNFPBfO5r?F@FuB^x{(3 zIN1Z%7ea5Zju)5jXo!9HUN2FxWRF5sv&r691R#N#=VRX@8<6X}-Zc>S)27-tp((~D zKiV&DmIMk6^#FXPnvX)ZnR=4uaozKIZT%Gl5WxPkFqFYnN-oYJT zW8>7|09vOVeD53l^~g=2!{_WRw8auDT-j$=YjhKOvX)F#`5n%Ua7e!e6R2MAJhdcL z2@mwsk%sh{I*Gj>)2M%X18vCYwb1v8%#d-O82c%!-|sqI9`ZhwEoljU4ChYy6K0%- zAQg)PGCm>b%TXq#We-;jK_>G+Q4bE74F(?T>wQGyeYqx-%Es{$;P`FqFO|n8&+Nv7 zpU|7K`J0^^75CkhD;c;TXVzS|>2+p^*%i`U7xcZK=WRgJHmhC(Jj%Pfe zyst55%Z(>FFdj7Ne`4TE`ttZk3dk-44Nm|lzcbl*C;$sHAv;LLX|-D!XfYnwdFMA1 z@>^!^1UxxR1=Y3iu+0-EdtO2^2U?MGwuTIwZXf}-;2}!(SBYC ztdGVdzdBbBC#NzYe0je)9M2?FPW9c;L7QKMI(+XP+uSZREm?%tfJ!sb{uGpSp%yHB zZ=jZmne%moD*TR!CPI9WXQ_>`AuJSW2VnYOsGKWWcnBR%V?*)FoIw5@>fz0Ptd|kz z>P4LTFD-cb&x|NHE=DhSWGefkf3FWkea@7BjOm5Jf71`&3gBZA3N$yK4r>t-aPS!B zu@qEQ3?F_|Fm4d!eA0q$Ao4W(_UUC+QzyFdqy>>Zjk9{~XBnHx5R$%3=c!rkrwI>3 zGT=Z_K$7PL=@?RNOecHD@#FDj{Dm(jt5D#){0)rUY^*QgE zdw%D^ggZGL&#KD%V(@rWH1Ylr*()eVxJrB@K7}TvJ=1gINR!LQh zQ+Yl*y_;BxXrKyJGiXD0^>TOOM-R{dsti7^za@-GVSD)U=r4)%P>4-;q#y zVyoJ@Dkh_i96&S%J|TdD1dl+*VbJT(fzxn$Q(!0Ze#9l zpg@b550l?wgyQOISY7)nki-_~XTkP@F7AXO?$AU+B$+Tr_WYRyN$tE9S6YT zf&9N0<3r5GC4ufAYE;U{YL5 z0)Ra5FB6<>IsF7)ZMJV(K9@(=#4e*vw|$E3KBpM*Ye4Wddc+NC9#$uV(Q&s`o| zP@#s6cutQU?0dvR1<$U%RIhd78hA1T*pu7)v;iYhvK%*Zw+vLE{bQLyROG`fPMcW2 zg`#_P`P%z@!iNooe|KQaLFJoFrKwABK956~|Bgx;76AwVsBHor=6H8|Zp?9&)45$~ z0F`K^Z!3Dm=vtjNhj$i6O$k6PLw_I(Jya_ga8~GL0UeRP=&H8RFzlVY+!b1mcO+5= ztWNIu5Gl+HB*)+^uZ&3S0Po21?&qsbpU3RwfvWKC%+76zQn!<#zMw3P#5pyfMoHmy$559{2KC3{ zPb+$)7mPgAEHJ^l*|9)woHG%%^k9E;Xx7B(}HmU7=!OM zTHgV)coMDlAYcG+Z+t7iOA?n>KC#~=eynG+OdqerKs)zpt`M5xM4>k!7bkEM5uRoA6m(HIi_In+lEvY9qgXi*rP`U zzQX5>>N0QW;u3c2v8GmkyY=!kRw~hpfk7JH6-F;vU`9?!gGy0z#=_jag9cRhpd(=Y z=sJSjX>S9zWgpCt0ATl5MTzl_w)n+Fd@U^LSJdhRCsalVj(AgZ>Z5aTDaYl*6@-7cmLMBM0MG6=o zb@~jKz#1VlLNJLjELazXupSIW7dy1Hfa`qze9hMTgrXS@;+|a!0U*FVk>cQXrr!k$ zci|8Jkj2Sz79ju{vaZhuzml|DHlMWcZ8-;aIFJm`Ob-zT?Z8tR8{zjLiIb;z+0#&L z;6o9{18MjiMnBYICu7tAq%a#16I*oZWNkZM?>iAPO{`BC0Y8v8DbN!nvesh#O;M>Y zy@z)_b_w;x2R|!;nPfcF=v7@dJZ)##4{E4!Gf70@J%=ulL0IAl6vyHK_!Ss!grZF; zAlCnovox()*Pj27yY^1rD|xPe;Qm2+C8>p3B0UGslp*a;Ow@$^PGO?VS20W*l>``aP0NKp?MH)T#q-$=phE*2HC#hh)Y-cptKaGxxnk; z*SX8H=z32G(&hZEb zUA0bd`x9}+VZ4(k<@6bY*lI@8Yv@gfqdyO*J$!nVF7i2e-V#95`D+L56@a0LpImr~ z5Olyc%8XSA;m@h^L6hFE)3mI?~3848WR~NRZP0A?1r|V|3K&bJ6On zW_!)AV{{ZR&t{NTSv+AP`zO^mhTw+o4eGdu}QerI61mn(|4QTT3b7Q3vW2DZib3 z?EgXbv4JaS(3A6z%3|KOuZT#8Lw%?=eoWjFdGW@RCtU)7k=*-x2~aP$u?by5lgdB( zg_(to3UrafA_?F{Pi8TVpi-ik0iau#8%g|glUMf?vc8M|hCh?lCPD_L=l30f!3US8 zxj?{zP6BOJH&S;2U~DfKeVgLWcn9eo@&wi2!h*Xg?j&LLG0*i7AUuQuB3mg5a;18X z7CGFa^6nD3xgHI$6!|%KXKD2CW9MT>^5pMGBLK-Yl?UcY536vB{aV8{{Ng2}AXkoq zEi@!BLt|vFj%MtsYA<%2f1j7b!YQzdX5n^U8|@z;e@DpuOO_2Cq7LF|{S)%)=3DOe zg`4tIKCaaF({lSbPH5+tVup1-t50iR&xqg3FaqP_4$yIcp2$&~ zfsR!Y$%jTgw1<1ZHT*AZc2a3s04F`(vE3`YA-TSaRWsh#L0;xhpr(VG(De~uQtXEL zVY0q{)@U`Alal>rXg=-@(D`x|$RK%F#q)h5jc~5wFrNcZz4Av7#3Q;3j5=@tM$dfi zKn$;~BTRjXE_s;$Pv3Inp}mhI9pqc7wU03Yq!7 zLpu-{G5`25$i)E#LOC4|!a+#+{RSy*l)Hd}?{$b^-Mw^}!HMB4{lkI1IUcAs;I^^h z853MY47d3482*?iH+=tZA|_38Gn5&h-XdEXWfWQZ`iKB16#4@69Beju{ckXFo(kuG zvx14KAxgy_h3PeVh4Dtu$j(46Hb}rzNMUTmYvMI-$IS1yhs#E};ROtqBb2o?&c`yO z-7PW!5T~MdHNX!%q``*=+|77y-F?#EO^M%!vWTyr!(3|fgU4%&9nl@p zK^Pzrdz(tX$UFaGz-NV}?k>|5WJ={Fksp}v1T#OqdCg05Bwh&pSq)%T9KcTlKC=J1 zJmWV!Fc(T=klOX>tONMP1ep9YJ*d!u{AZL#inu~Ykfc4-ih-e6VE}`4X{|x9xY+2@ z%de~{_;1)74uC!AiLmEjrMOaEh`%;SxsRlleJV(7;Ja50cZOQO?@u~AYJosWGId)G zR%VODl;N#)fmg#M^ZzU2%;TYKyEtwcS$i1XY>_P4B186sB1-$WoRO(O3pU zlYPxZ*%Fb-t|E-JiDa54lYI#lLYAkP=NjJUegB@%=bHOI=Q`)y*SXI3cOcyXQcD+w zA7vpz2x_&3nd&H|;H$%O`N`}B0GK`Yw-@j9H!w#nDuVE{&{4Yb2Jz}LsO8)7m)qWS zX}fmm3sqZ(>@(lZQ=`zl1euAW-10vL-~JC(cNWQDR-*@50^ne}2FW(;<;TAasIwaOcyn|#sXvf73-54X#QH~Q zD~x!8jXd3jeOOYPFYUrqV_|89cR5DtC$bl1_tXy&VA7TiRRdAjD68On{rCRj1fx6>d}4?uWnZbVy4 z3Q{=`ElsDzXNBj*IR8tX!+#QOrHv}!yh?t*pdXXqyMzOCFxe-Qg|9Ab^XO8REUnp< z=n@wg?o5_}M{N1896x|DWpl}T)7+C$oa-%q!a4u0H$$oX$Soqw3$7MDn?_-9T_R;r zefq@6J`OK{WJ23Zj7td1OWXt$}z1s|^K&=1hE;HcCU)TT6(|dvm zCl)E?>(X5hessmE(GpS?5ai)tW#fn~Zy_p}vt<`dTA@E&stx2VGz|*#DH!uL0M!PB zffLXt817YZyZ=aP-kKh77)QZD3VF@*C9r-`W1zNclZEq+-I1@{YHyu&RKO->6F1S+ z$ie$hPBH*}4-m93>t9Iipc{xqCl4YCP2}+U^46n?8NhNJfPfdQ6zl;UD;+nx4Wjze zK~$qdi6sf3n7EQhm{K!s+Z~4}{Gc%qvq-$NirEiX1_q$8bKSAbn_S?disS`CL15r( zpDXA4A6ITEQ^E6EA{qaB`JIt{ZyxvE4K-U@C9_|q(gGMW1`T$rNlYXk*a>~gpPM{j zjo>g}XHm41@mNy3K41yqcX{^$uSZPCpJX0-icewBKZ32*A|8?Ub*^$h8$^(ye3uR1 zhRw)=29o6NC_E-^Z9gf=^tEX{#J}@u1S&m}Ohz-|5A^)ICSB({`}zf>^@XN@RL4QC zgjktLf)g14@$XzydJq|sp7?kSV}Ppi@XLK#dLSS|Kb-W*aqMMr;&D5hHy4!Vz~CVi zCb3oL;0G=%c#M@Tz@p)b$3p?hyc% zyMy{c&)}6F;FFvIEX@}89mrtl8TC_Yszf`+C$eGW!#WRDL6;k}mGjPEF35sB?j|>8 z9dA^mDehryjgp(!J9|`ueciu!{dYi?uHz{Lyz|?5O?Ls3~J9@+X(c7vDC#i%uM37(`*it&sHOY4JiHzl8p@^_}fVi(2G zbEC6Tq7&S+KZ;g4A#Jm`6&fnRI)G{1{7+P7^Y)VdsC4mhj)m=d(Ttg?C$vlzK{vy! z^ucFcBd;I6CwYIU2}Z4Sm)XQ&6axnNRx(W|m+ekD>P8)|!jp1&7$@JbR@WZG%c=^M zb@E6SKZ9$Z(D@b0YWO-2IYub5S-7)ze$+;Bz)DA9W?=D#p!OLY{#MHtDS#BFl_iAz z;D&)(k+Id(_n$`!k&>zm3YQFTP0g7#)9&A~fv+;h)h(Ra6X-z|^ffoOy^Wb~vY(X| z=oBcinWJ$^7Qavlo^#m^LF$W`)?@7g}bHev2Pjqb`) z#q7}c7z2^7P*|BxO`Ox!_=tf z6T54uxK!BSsc_E^ep3~VU!PgsQ%%TjA<|xv{TmjCe}3kJHReo|Szhe|Ha=`79W(4K z5O7x~V>9Gl;xOFq(PrKeDtCeFqUyODq=4IngaLW4=R>X$#~ElVnf4*^L3>=(VNuNx zq@KL|4thSQ2domC-`)`2KG7btGlg(o5r91M4O7D^s-@w0D{;J~PP`k36iFGvvD-AG zc#$N@V>!^Z3p{HH&^0vvW+1c<%v9;Ry4aMQqvm^QS}c3Sxbq=^E6<(aGi`4BHH{oH zx$>ji5fcF|He!*oT0Q(RmVx3@VL;9AeXLeGLAo>-2kdwOuiAj6o()@_6wtde)8ga? zf3O(Ms2I2szmtY1l~|?#d%@vQ)PJ__IIZH=#qrk8u3p;SghCBrMASUqI`w=w*gxqK z5__^=Cc>4GcSVNtzc2mhKJIH5i?5M2L~1 uR3f@Ef;~IsB3#YneST=!o_1+8je$dfv(P;3%o0$vV=%p7VOVn>5&I9wQmQEc literal 0 HcmV?d00001 diff --git a/docs/source/conf.py b/docs/source/conf.py index 5c680d97..d9ab4e48 100644 --- a/docs/source/conf.py +++ b/docs/source/conf.py @@ -57,9 +57,17 @@ # Path to generate documentation from using sphinx AutoAPI autoapi_dirs = [os.path.abspath(os.path.join("..", "..", "causal_testing"))] +autoapi_generate_api_docs = True +autoapi_keep_files = True + +# Suppress label warnings +suppress_warnings = ['autosectionlabel.*'] + + html_logo = '_static/images/CITCOM-logo.png' html_theme_options = { 'style_nav_header_background': '#9ADBE8', # Set the colour using CSS 'logo_only' : True, -} \ No newline at end of file +} + diff --git a/docs/source/frontends/json_front_end.rst b/docs/source/frontends/json_front_end.rst index a18a389f..7d0fe957 100644 --- a/docs/source/frontends/json_front_end.rst +++ b/docs/source/frontends/json_front_end.rst @@ -20,7 +20,7 @@ An example is provided in `examples/poisson` which contains a README with more d run_causal_tests.py ******************* -`examples/poisson/run_causal_tests.py `_ +The `examples/poisson/example_run_causal_tests.py `_ contains python code written by the user to implement scenario specific features such as: @@ -34,9 +34,9 @@ Use-case specific information is also declared here such as the paths to the rel causal_tests.json ***************** -`examples/poisson/causal_tests.json `_ contains Python code written by the user to implement scenario specific features -is the JSON file that allows for the easy specification of multiple causal tests. Tests can be specified two ways; firstly by specifying a mutation lke in the example tests with the following structure: -Each test requires: +The `examples/poisson/causal_tests.json `_ contains Python code written by the user to implement scenario specific features +is the JSON file that allows for the easy specification of multiple causal tests. +Tests can be specified two ways; firstly by specifying a mutation lke in the example tests with the following structure: #. name #. mutations @@ -57,17 +57,22 @@ The second method of specifying a test is to specify the test in a concrete form #. expected_effect #. skip + +Alternatively, a ``causal_tests.json`` file can be created from a ``dag.dot`` file using the ``causal_testing/specification/metamorphic_relation.py`` script as follows:: + + python causal_testing/specification/metamorphic_relation.py --dag_path dag.dot --output_path causal_tests.json + Run Commands ************ This example uses the ``Argparse`` utility built into the JSON frontend, which allows the frontend to be run from a commandline interface as shown here. To run the JSON frontend example from the root directory of the project, use:: - python examples\poisson\run_causal_tests.py --data_path="examples\poisson\data.csv" --dag_path="examples\poisson\dag.dot" --json_path="examples\poisson\causal_tests.json + python examples\poisson\example_run_causal_tests.py --data_path="examples\poisson\data.csv" --dag_path="examples\poisson\dag.dot" --json_path="examples\poisson\causal_tests.json A failure flag `-f` can be specified to stop the framework running if a test is failed:: - python examples\poisson\run_causal_tests.py -f --data_path="examples\poisson\data.csv" --dag_path="examples\poisson\dag.dot" --json_path="examples\poisson\causal_tests.json + python examples\poisson\example_run_causal_tests.py -f --data_path="examples\poisson\data.csv" --dag_path="examples\poisson\dag.dot" --json_path="examples\poisson\causal_tests.json There are two main outputs of this frontend, both are controlled by the logging module. Firstly outputs are printed to stdout (terminal). Secondly a log file is produced, by default a file called `json_frontend.log` is produced in the directory the script is called from. diff --git a/docs/source/index.rst b/docs/source/index.rst index c083e96f..88985895 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -4,11 +4,12 @@ Welcome to the Causal Testing Framework |status| |ci-tests| |code-cov| |docs| |python| |pypi| |doi| |license| Overview -******* +********** Causal testing is a :term:`causal inference`-driven framework for functional black-box testing. This framework utilises graphical causal inference (CI) techniques for the specification and functional testing of software from a black-box -perspective. In this framework, we use causal :term:`directed acyclic graphs` (DAGs) to express the anticipated cause-effect +perspective. In this framework, we use causal directed acyclic graphs (DAGs) to express the anticipated cause-effect +relationships amongst the inputs and outputs of the system-under-test and the supporting mathematical framework to relationships amongst the inputs and outputs of the system-under-test and the supporting mathematical framework to design statistical procedures capable of making causal inferences. Each causal test case focuses on the causal effect of an intervention made to the system-under test. That is, a prescribed change to the input configuration of the @@ -118,7 +119,7 @@ system-under-test that is expected to cause a change to some output(s). :hidden: :titlesonly: - /autoapi/causal_testing/index + /autoapi/index .. toctree:: :hidden: @@ -155,7 +156,7 @@ system-under-test that is expected to cause a change to some output(s). .. toctree:: :hidden: - :maxdepth: 1 + :maxdepth: 2 :caption: Credits credits @@ -192,4 +193,4 @@ system-under-test that is expected to cause a change to some output(s). .. |license| image:: https://img.shields.io/github/license/CITCOM-project/CausalTestingFramework :target: https://github.com/CITCOM-project/CausalTestingFramework - :alt: License \ No newline at end of file + :alt: License diff --git a/docs/source/installation.rst b/docs/source/installation.rst index 7e6c07a8..48736aa3 100644 --- a/docs/source/installation.rst +++ b/docs/source/installation.rst @@ -1,14 +1,14 @@ Getting started -============ +================ Requirements ------------- +--------------- * Python >= 3.9. * `Microsoft Visual C++ `_ 14.0+ (Windows only). Installation ------------ +----------------- The Causal Testing Framework can be installed through either the `Python Package Index (PyPI)`_ (recommended), or directly from source. .. _Python Package Index (PyPI): https://dl.acm.org/doi/10.1145/3607184 diff --git a/docs/source/modules/causal_specification.rst b/docs/source/modules/causal_specification.rst index 083a5a5b..9774e019 100644 --- a/docs/source/modules/causal_specification.rst +++ b/docs/source/modules/causal_specification.rst @@ -40,4 +40,4 @@ there are a couple of requirements that should be satisfied. - Collectively, the components of the causal specification provide both contextual information in the form of constraints and requirements, as well as causal information in the form of a causal DAG. Later on, these components will be used to design statistical experiments that - can answer causal questions about the scenario-under-test, such as ``Does opening a window impair the viruses ability to spread?`` + can answer causal questions about the scenario-under-test, such as `Does opening a window impair the viruses ability to spread?` diff --git a/docs/source/modules/causal_tests.rst b/docs/source/modules/causal_tests.rst index e8207c19..a8ea33c9 100644 --- a/docs/source/modules/causal_tests.rst +++ b/docs/source/modules/causal_tests.rst @@ -42,9 +42,6 @@ We then define a number of causal test cases to apply to the scenario-under-test * ``mask_wearing_test = (X={precaution = None}, \Delta = {precaution = Mask}, Y = {-20% < n_infected_t5 < -10% })`` (Mask wearing is expected to result in between 10% and 20% fewer infections). * ``hand_washing_test = (X={precaution = None}, \Delta = {precaution = Hand Washing}, Y = {-40% < n_infected_t5 < -25%})`` (Hand washing is expected to result in between 25% and 40% fewer infections). -Data Collection ---------------- - - To run these test cases experimentally, we need to execute both ``X`` and ``\Delta(X)`` - that is, with and without the interventions. Since the only difference between these test cases is the intervention, we can conclude that the observed difference in ``n_infected_t5`` was caused by the interventions. While this is the simplest approach, it can be extremely inefficient at scale, particularly when dealing with complex software such as computational models. - To run these test cases observationally, we need to collect *valid* observational data for the scenario-under-test. This means we can only use executions with between 20 and 30 people, a square environment of size betwen 20x20 and 40x40, and where a single person was initially infected. In addition, this data must contain executions both with and without the intervention. Next, we need to identify any sources of bias in this data and determine a procedure to counteract them. This is achieved automatically using graphical causal inference techniques that identify a set of variables that can be adjusted to obtain a causal estimate. Finally, for any categorical biasing variables, we need to make sure we have executions corresponding to each category otherwise we have a positivity violation (i.e. missing data). In the worst case, this at least guides the user to an area of the system-under-test that should be executed. diff --git a/docs/source/usage.rst b/docs/source/usage.rst index 842fe3f1..7f1e701d 100644 --- a/docs/source/usage.rst +++ b/docs/source/usage.rst @@ -103,5 +103,4 @@ various information. Here, we simply assert that the observed result is (on aver test_passes = causal_test_case.expected_causal_effect.apply(causal_test_result) assert test_passes, "Expected to see a positive change in y." -Multiple tests can be executed at once using the test engines `test_suite `_ -feature. +Multiple tests can be executed at once using the test engines :doc:`Test Suite ` feature.