📢 Upcoming Release of v3.0.0 schema: Enabling formal sample output type support

[annakrystalli]

In the near future we will be releasing schema version v3.0.0 which contains breaking changes affecting a number of packages across the hubverse.

These changes will only break functionality in existing hubs that collect forecasts using the sample output type.

Support for sample output types in tasks.json

The most important breaking change involves changes to the specification of sample output types (See Sample Output Type Specification Doc for more details).

We are adding more formal support for sample output types. The main difference to previous schema and handling throughout the hubverse is that, instead of configuring samples in the same manner that is used with other output types, via an output_type_id object containing vectors of required and optional values, the sampling regime is defined via an output_type_id_params object instead., e.g.:

"sample": {
    "output_type_id_params": {
         "is_required": true,
         "type": "integer",
         "min_samples_per_task": 50,
         "max_samples_per_task": 100,
             "compound_taskid_set": ["forecast_date", "target", "horizon"]
    },
    "value": {
        "type": "integer",
        "minimum": 0
     }
}

Implications for hubverse packages

The main area where the breaking change in the schema will cause breaking changes in package functionality is in creating sample output type objects in hubAdmin.

Back-compatible updates will enable support for v3.0.0 sample specifications when:

• Validating tasks.json config files in hubAdmin.
• Creating Hub schema in hubData.
• Creating model output templates and accepted value grids in hubData.
• Validating submissions with sample output types in hubValidations.

To access the updated functionality you will need to install the latest versions of packages once released.

In time, additional new functionality for supporting and working with sample output types will be added to hubData, hubVis, hubEnsembles and hubEvals as well.

These changes only affect hubs that are collecting sample output types. If a hub is collecting samples and using a tasks.json config with a schema_version earlier than v3.0.0, breaking functionality will error and advise users to upgrade their config to conform to version v3.0.0.

Introduction of repository object in admin.json

Another breaking change introduced in schema version v3.0.0 is that the repository_url and repository_host properties in admin.json have been deprecated in favour of a single repository object with separate host, owner and name properties.

This change is aimed at making cloud hosting easier to configure. It does not affect any package functionality but it does mean that switching to using v3.0.0 schema, e.g. in order to use the new sample output type config, will also require updating your admin.json config file to conform to the new repository specification.

Release plan & Further updates

We are planning on rolling out the changes in the schema and hubverse packages at the same time. While an exact release schedule is not known yet, it will likely be in the next 3-4 weeks.

In the meantime, you can follow this announcement for further updates and follow progress/test out functionality in associated PRs in the relevant repos/packages.

[M-7th]

Thanks @annakrystalli, that's great news.
Since we are planning to use the sample output type in the new European Scenario Hub we are starting to work on, we were wondering if there's a way to set up a test environment to try this new schema before the official release.
thanks for your help

[annakrystalli]

Test driving v3 sample output types

More details on the new sample output type specification can now be found in our hubDocs.

The best way to test drive new feautures would be to use the v3.0.0 schema version PR that is queued up and install development versions of packages from development PRs/branches. Note that any branches with [WIP] PRs should be considered experimental, i.e. they have likely not been reviewed, may contain errors or may change.

• v3.0.0. schema PR: add v3.0.0 schema with initial sample schema schemas#72
• hubAdmin PR (Reviewed): Support v3.0.0 schema sample specification hubAdmin#19
  You could use hubAdmin::validate_hub_config(hub_path = hub_path, branch = "br-v3.0.0", schema_version = "latest") to check that your hub will pass v3.0.0 schema validation.
• hubData PR (somewhat stable, my change): Support v3.0.0 schema sample specification hubData#31
• hubValidations PR: Feature/ Handle V3 sample specification hubValidations#82
• hubUtils PR: Enhancement/v3 utils hubUtils#148

You can use the ref argument to install from a specific branch of a remote, e.g.

remotes::install_github("Infectious-Disease-Modeling-Hubs/hubAdmin", ref = "feature/sample-support")
remotes::install_github("Infectious-Disease-Modeling-Hubs/hubData", ref = "feature/handle-samples")
remotes::install_github("Infectious-Disease-Modeling-Hubs/hubValidations", ref = "feature/handle-samples")
remotes::install_github("Infectious-Disease-Modeling-Hubs/hubUtils", ref = "enhancement/v3-utils")

Note about test driving hubValidations

If you'd like to test out validating v3 configured sample model output files, you'll need to use

validate_submission(
  hub_path, file_path, ...
  skip_check_config = TRUE
)

otherwise you will get an error about a non valid config version (as technically v3 is not available yet on the main branch).

[M-7th]

Thank you very much @annakrystalli! This is very helpful

[annakrystalli]

Announcing Important upcoming updates to the hubverse

Week Starting 17th June 2024

In the week starting 17th June 2024, we'll be rolling out two important changes that have been in the works for some time now:

1. Release of v3.0.0 schema and updated versions of packages to support more formal handling of sample output types.
2. Changing the GitHub organisation name from Infectious-Disease-Modeling-Hubs to hubverse.

Timeline

Monday 17th June - Organisation Name change to hubverse

Wednesday 19th June - Release of v3.0.0 schema and new versions of packages.

If you are subscribed to our mailing list you will receive an email reminder the day before each change.

Details

More details on the new sample output type specification can now be found in our hubDocs.

The new schema also introduces a number of changes/new features:

• The repository_url and repository_host properties in admin.json have been deprecated in favour of a single repository object with separate host, owner and name properties.
• The optional hub_models property in admin.json has been removed as it's schema could create conflicts with the model_abbr and team_abbr schema specification in the hub administered model-metadata-schema.json.

• Additional properties are now allowed at the round item property level in tasks.json. This was at the request of hubs who wished to be able to store additional metadata at the round level. Note that the strictness of not allowing additional properties meant that misplaced objects in the config would result in a config validation error previously. This is not the case anymore at the round item level, so do take care with the objects you place in the round property of your config.

Implications for Hub administrators

None of these changes should affect the normal functioning of hubs. The vast majority of changes to packages are back-compatible and if hubs are not interested in collecting sample data, there is no need to upgrade to the v3.0.0 schema.

You do need to upgrade to v3.0.0 schema if you:

1. wish to collect or prepare your hub for collecting sample output types.
2. wish to configure your hub for cloud mirroring using the new repository object in admin.json.
3. wish to store additional metadata at the round level in tasks.json.

With respect to the organisation name change, URL re-directing means that the change should not affect functionality of current versions of packages. At some point soon after the name change, however, hub admins would be advised to perform a Find and Replace and replace the old name (Infectious-Disease-Modeling-Hubs) with the new org name (hubverse) throughout their hubs. Also the hub remotes will need to be changed.

NOTE THAT NO ACTION IS REQUIRED UNTIL THE RELEASE OF THE v3.0.0 SCHEMA AND NEW PACKAGE VERSIONS ON WEDNESDAY 19th JUNE.

Admin Task List

Must be performed after organisation name change & new package versions release

• Find and replace Infectious-Disease-Modeling-Hubs with the new org name hubverse (How to do this in Rstudio and VS Code.)

If upgrading to v3.0.0 schema

• Change version in the schema_id property in admin.json and tasks.json to v3.0.0.
• In admin.json, change repository_url and repository_host to new repository object.
• In admin.json, remove hub_models object. If you need to store such information, consider encoding it as part of model metadata.
• Install newest versions of any packages used.

Use newest version of hubAdmin::validate_hub_config() to ensure your changes to the schema are valid.

Test driving new functionality

The best way to test drive new features would be to use the v3.0.0 schema version PR that is queued up and install development versions of packages from development PRs/branches.

• v3.0.0. schema PR: add v3.0.0 schema with initial sample schema schemas#72
• hubAdmin PR (Reviewed): Support v3.0.0 schema sample specification hubAdmin#19
  You could use hubAdmin::validate_hub_config(hub_path = hub_path, branch = "br-v3.0.0", schema_version = "latest") to check that your hub will pass v3.0.0 schema validation.
• hubData PR (somewhat stable, my change): Support v3.0.0 schema sample specification hubData#31
• hubValidations PR: Feature/ Handle V3 sample specification hubValidations#82
• hubUtils PR: Enhancement/v3 utils hubUtils#148

You can use the ref argument to install from a specific branch of a remote, e.g.

remotes::install_github("Infectious-Disease-Modeling-Hubs/hubAdmin", ref = "feature/sample-support")
remotes::install_github("Infectious-Disease-Modeling-Hubs/hubData", ref = "feature/handle-samples")
remotes::install_github("Infectious-Disease-Modeling-Hubs/hubValidations", ref = "feature/handle-samples")
remotes::install_github("Infectious-Disease-Modeling-Hubs/hubUtils", ref = "enhancement/v3-utils")

Note about test driving hubValidations

If you'd like to test out validating v3 configured sample model output files, you'll need to use

validate_submission(
  hub_path, file_path, ...
  skip_check_config = TRUE
)

otherwise you will get an error about a non valid config version (as technically v3 is not available yet on the main branch).

Keeping up to date

For further updates, feel free to follow this GitHub Announcement and reach out with any questions here.