Schema failures on message format 2 in iCU76

[sven-oly]

No description provided.

[sven-oly]

@mradbourne
I see this error message in icu76 generated test files for message_fmt2.

  File "/usr/local/google/home/ccornelius/ICU_conformance/conformance/testgen/generators/message_fmt2.py", line 54, in process_test_data
    logging.error("Problem validating JSON: %s agains schema",
Message: 'Problem validating JSON: %s agains schema'
Arguments: ('/usr/local/google/home/ccornelius/ICU_conformance/conformance/testgen/generators/../icu76/message_fmt2/message-format-wg-tests/functions/datetime.json', PosixPath('/usr/local/google/home/ccornelius/ICU_conformance/conformance/testgen/generators/../../schema/message_fmt2/testgen_schema.json'))
[2025-01-09 15:53:02,778] [ERROR]  Additional properties are not allowed ('$schema' was unexpected)
Failed validating 'additionalProperties' in schema:
    {'$comment': 'This allOf specifies required test properties that allow '
                 'a default. A value will be required in '3. 

                 "'defaultTestProperties' if one is not provided for every "
                 'individual test.',
     '$defs': {'anyExp': {'anyOf': [{'required': ['exp']},
                                    {'required': ['expParts']},
                                    {'required': ['expErrors']}]},
               'bidiIsolation': {'description': 'The bidi isolation '
...
     '$id': 'https://raw.githubusercontent.com/unicode-org/message-format-wg/main/test/schemas/v0/tests.schema.json',
     '$schema': 'https://json-schema.org/draft/2020-12/schema',
     'additionalProperties': False,
...
     'required': ['tests'],
     'title': 'MessageFormat 2 data-driven tests',
     'type': 'object'}
On instance:
    {'$schema': 'https://raw.githubusercontent.com/unicode-org/message-format-wg/main/test/schemas/v0/tests.schema.json',
     'defaultTestProperties': {'expErrors': False, 'locale': 'en-US'},
     'description': 'The built-in formatter for datetimes.',
     'scenario': 'Datetime function',
     'tests': [{'exp': '{:datetime}',3. 

                'expErrors': [{'type': 'bad-operand'}],
                'src': '{:datetime}'},3. 

...

This is one of the generated test files that includes "$schema" property, but the file testgen_schema.json doesn't allow that as a property. I think all of them are actually failing schema validation

Elango and I see two options:

1. Allow additional properties
2. In the list of properties, add "$schema".

There may be other options, but right now I think most or all of the mf2 validations are failing.

What do you think?

[mradbourne]

I think it's useful to add "$schema" to the list of properties, so option 2.

In future, we'll likely want MessageFmt2Generator to handle a range of source formats with different schemas (e.g. ICU77 MF2 tests might need to be parsed differently to ICU76). We can use the $schema property present in the source JSON to tell MessageFmt2Generator how to validate and parse it.

I can raise a PR for this.

[sven-oly]

Hey Matt, thanks for your note on this. I've implemented another option in PR #394, i.e., removing the "$schema" part as the tests for conformance are generated.

I tried option 2, naively adding "$schema" as another property, but the schema validation didn't like that as another part of the schema. There may be way to do this, but I'd like to do this PR for now.

Also, I see that there are two schema files: test_schema.json and testgen_schema.json. Do you know why there are two?

Thank you again for your help with this part of Unicode Conformance.

[sven-oly]

#394

[mradbourne]

Also, I see that there are two schema files: test_schema.json and testgen_schema.json. Do you know why there are two?

@sven-oly Yes, testgen_schema is the input to the test generator.
test_schema is what the executors expect as input (i.e. the output of the test generator).