Metadata to describe the config itself: status:draft|final, warnings, general information, etc.

[pierrot0]

A Croissant config describes a dataset, so fields like description are about the dataset.
However there are no fields to describe the config itself.

Topic came up in #45 (comment): the PR introduced a config and a README.md file next to the config to precise that the config was not describing the dataset fully. A comment was rightfully made that the warning could go in the description field of the config, as not to scatter the information.

The problem with that approach is that it mixes data about the dataset and the config itself, and so I thought it might be handy to have metadata fields describing the config itself, such as for example the status (draft, ready for consumption, config version?), or just general meta descriptions, todos, warnings, etc.

What is the sentiment around such meta metadata?

[pierrot0]

Such a "config-specific metadata" could all be specified in a croissant subsection:

{
  ...
  "@type": "sc:Dataset",
  "name": "dataset_name",
  "croissant": {
    ...
  }
}

What I think could be useful in this croissant dictionary:

• croissant_version: string, the croissant version the config follows.
• snapshot: boolean, true if the config allows to load data reproducibly (all fileItems have a checksum provided).
• description: a general description of the config, where we put information about known limits (eg: "config only partially describes the dataset, are missing: X, Y, Z") or other information worth knowing for the consumers or maintainers of the config. Maybe we want to be more precise than that and have:
  • todos: list[string], TODOs relative to the config (as opposed to the dataset).
  • warnings: list[string], warnings relative to the config.
  • etc.

At some point I think it also makes sense to keep a version of the config itself and a changelog. However maybe such information would be better fitted outside of the config itself (CHANGELOG.md + version file in github?).

Please let me know what you think.

[ccl-core]

+1 separately storing croissant-related info in a metadata dictionary would be much cleaner, and keeping it in the config itself is the solution that at this stage makes more sense to me!

Following #91 (comment), the croissant source could also be a field in the croissant metadata dictionary.