Hub Validations Scope

[annakrystalli]

Hub repositories: https://github.com/Infectious-Disease-Modeling-Hubs

3 levels of validation:

Files on the way in

• File locations, names, etc
• File contents consistent with hub configs
• State of a hub

Configs are correct

• DONE: checks for individual files
• TODO: checks for individual files, for remote data. Issue has been filed
• DONE: overall check for all hub config files. Issue has been filed
  Q: should this functionality move to hubValidation? Currently in hubUtils
  A: yes

Data in hub consistent with configs

• Depends on 1 a, b

Schema validation,

So that when the consortium publishes schema, they are correct
Code in hubValidation, schema, or hubDev?
Working decision: hubValidation

Tasks:

• Scope out what validations are required by each group
• Develop validation workflow visualization (flowchart?)
• Create new repos for :
  • hubValidations
  • hubCI that will be R packages
• Create testing repository with example submissions (good and bad examples of forecasts and scenario projections)

General thoughts about architecture/framework

hub-validations.json metadata file that contains info about which validations to implement in a CI setting:

• Complement to the other config files by defining overall structure of what validations are run
• Store in the hub-config folder
• Validation tasks are fixed and defined as below
  • validate_submission
    • Run for for new/updated files in one submission
    • Collection of the following task groupings
      • validate_model_metadata
      • validate_model_data_file
      • validate_submission_time (things specific to PRs, date alignments, etc…)
    • validate_repository
      • Run for all models, files
      • Collection of the following task groupings
        - validate_model_metadata
        - Validate_model_data_file

The specific validation groupings are defined inside hub-validations.json

{
	validate_model_metadata: {
	‘default’: [
		{
		fun: ‘hubValidations::validate_metadata_file_name’,
				args: {}
},
		{
fun: ‘hubValidations::validate_metadata_…’,
args: { ‘arg1’: ‘value’ }
},
…
	]
},
	validate_model_data_file: {
	‘default’: [
		{
		fun: ‘hubValidations::validate_data_file_name’,
				args: {}
},
		{
fun: ‘hubValidations::validate_data_file_task_ids’,
args: { ‘arg1’: ‘value’ }
},
…
	]
	},
	validate_submission_time: {

	}

hubValidations package:

Scope is to write many functions that each address one test specified on the validations list. All tests implemented by this package are for model output submission file contents, file name, and placement of file in a hub structure.

Possibly define a generic “validate” function

Arguments:

• test object: file or data structure?
• criteria: model file or line/chunk/property from metadata file
• comparator: how well, boolean or score, does the test object conform to criteria? Diagnose failures - give array of error msgs

Or separated in multiples functions:

• "Content": test object: model projection file content to test against the "config" file (with the possibility to add hub specific tests) (wrapper of multiple functions)
• "Structure": test the filename, fileformat, path, etc. (use information from the "config" and "admin" files
• "Metadata": test the associated metadata file format (or abstract?) against a schema available in the hun

Acquire information from the hub metadata files (possibly via hubUtils hub_connection function)

E.g. Specification of columns and values in the data files, format of the columns, etc.

Implement functions for specific tests that have standard

• Input
  - files/data objects from environment
  - Hub metadata specifications
  - Auxiliary data files (or pointers to such files if not in hub metadata)
• Output: named list
  - Name: Name of the test
  - Result: Boolean validation result. Expect this to be TRUE if there were no errors or warnings, FALSE if there were any errors or warnings
  - Errors: Collection of errors, warnings, and informational messages

Open questions

• How many errors is one test allowed to generate? 1 or more than 1?
• What data structure used to store these? Options:
  • As part of above output list, entries errors = …, warnings = …, messages = …
  • A single list(?) of errors, warnings, messages
  • What type of object? rlang::error_cnd, rlang::warning_cnd, rlang::message_cnd
• How should we track whether an error means some later steps should not be run?

Collect results from multiple specific tests into a list

Consider implementing a concept of a group of tests

Idea being that common
Example functions:

• Check forecast submission date
• Check filename
• Check locations
• Check for negatives, larger than population, etc.
• Check for valid/full set of quantiles
• Check for acceptable prediction types

“hubCI” package:

scope is to wrap tests from hubValidations for integration in a CI server, and implement validation steps that are specific to storage of files in version control.

Acquire information from the hub metadata files (possibly via hubUtils hub_connection function)

• Specification of validation steps to run

Acquire information from a pull request with model output submissions

Run each validation step, collect outputs

Output report, messages

Example tests

• Does a submission file update an existing file?

Process thoughts

• Review European Hub R package for validations to see if there is any useful code there that could be ported.
• Start with basic functionality that would be used across a broad set of Hubs
• highlight/identify fields in sheet of desired validation functionalities of different types to see what would be need in the infrastructure
  Longer term?: retain ability for hubs to write their own hub specific validation functions (say in R) that could be combined with a wrapper framework

Validation Process

1. Workflow initiation
   a. Local validation
   b. Validation on PR
2. Validation setup
   a. Use either:
   1. Input variables, or
   2. Metadata
   b. Define validation structure (flow of function calls and validation checks)
3. Run validation
4. Output validation results

Scope:

• Validating correctness of submissions:
  • File location/name
  • File contents – technical structural (e.g. column names correct)
  • File contents – valid forecasts (e.g. increasing quantiles, integer non-negative values)
  • File contents – forecasts look reasonable?
    • e.g. population constraints
    • E.g. level, slope match recent data
  • Submission timing?
  • Associated metadata / abstract?
• Ensemble construction – which models are valid for using in an ensemble
• Validation of arguments to functions
  • If I’m loading forecasts and I want to specify a set of locations to load, did I specify valid options for locations

For today: just the first point, validation of submissions?

What is required:

• Definition of standards and rules
  • What are the validation rules that are implemented?
  • How are the checks that are done for a hub specified in the json config file?
    • Different checks done for different hubs
    • Potentially different outputs for the same check (e.g. error vs warning)
• Packages implementing validation checks
• Workflows (e.g. GitHub actions)
• Documentation of all of the above
  • Could think of this as a tutorial for how to set this up using these tools

Validation development details/plan

• Scoping of validation checks
• List validation checks: https://docs.google.com/spreadsheets/d/1GgA4KgCsuEkAGBJcWNIXf_c3uqq9rsKn-pXzNBFxFuQ/edit#gid=0
  Checks should be cross-referenced with elements in JSON file

Current practices

SMH hub validations,

Repo: https://github.com/midas-network/SMHvalidation

• has basic functionality for reading JSON
• Will be in use for a covid round 17 "megaround" in April 2023
• Maybe at some point when the time/need arises, someone would use this repo as a base for Infectious-Disease-Modeling-Hubs/hubValidations R package

US Scenario Modeling Hub (covid, flu):

R package: https://github.com/midas-network/covid19SMHvalidation

• Contains a function that runs multiple checks and outputs a long message (“report”) with any errors or warnings.
• All checks are written in a vignette (validation-checks).
• Contains also a visualization function that print a pdf with all the projections per target/state with or without observed data
• The package is currently going through a complete rewriting to be able to be used on both FLU and COVID SMH (progress on the branch flu_update)
• Include processing a json containing the expected format of each submission (hub metadata files)
• The package should be called automatically and run both checking, visualization functions on each PR with a submission file. (only running automatically for COVID)

Limitations:

• It currently does not contain any code for testing the metadata, abstract, folder name associated with the Submission
• Some tests/codes are highly specific and will need rewrite to adapt to global format but it’s possible

US Forecast hubs (covid, flu, west nile virus)

Validations are implemented in python: https://github.com/reichlab/covid19-forecast-hub-validations
Documentation: https://docs.google.com/document/d/1OAL2pcWmfssJlE6wIbV3PduU689ZctKJNFGv257W0Vk/edit#heading=h.o8oxo12w6bl8

• Validations run automatically upon submission using GitHub actions
• Results are posted to the PR as comments
• There is a reasonably thorough set of unit tests, described here: https://docs.google.com/spreadsheets/d/1TVe45VsBTMCkyZFiZm3v6O5AGN5q5edhg3AQXZVe-mw/edit?usp=sharing
• At a high level, three types of validations are performed:

Files modified:

• File location and name correct
• only allowed files are modified

File contents are correct. Things like:

• Model output files:
  • Correctly formatted csv file, correct column names
  • Values in columns are correct
  • Predictions of incidence can’t be negative or greater than the population size
• Model metadata files:
  • Pass validation against the format specified by metadata schema
• Submission timing is OK
  • Per-hub specification of allowed forecast dates
  • Per-hub specification of a window around the forecast date within which the PR must be submitted

Limitations:

• Currently there are some settings hard-coded; adding support for a new hub requires some manual edits to the code.
• No forecast viz upon submission.
• Very much tied into GitHub as the submission platform.
• Some more work would need to be done to complete the unit test suite and to convert some informal “tests” into more formal/reproducible unit tests.
• It’s generally our feeling that the code base could use some cleanup; we’ve never really had a senior developer who “owned” this project.

EU Hubs:

Python library, inherited from the DE/PL hub, who inherited it from the US flu forecast hub. However, this succession of maintainers, and successive patches done in a rush made it completely unmaintainable. It took us months to even realize that some tests were not running.

As a reaction, we developed the HubValidations R package: https://github.com/covid19-forecast-hub-europe/HubValidations

• Matches the forecast data and metadata to a json schema
• Check file locations & names

Pros:

• Very flexible and works well with the quite different formats from the EU forecast & scenario hubs.
  100% configurable by users. Changes in formats shouldn’t require changes in the package code
• Can run locally or on CI
• Can run at different levels: single data file, single metadata file, single model, complete hub

Cons:

• Difficult / impossible to set up more advanced tests, i.e., tests that involve data transformation (operation on a single or multiple columns)

Zoltar

Tests for the Zoltar repository: https://github.com/reichlab/forecast-repository/tree/master/forecast_app/tests