How to migrate from RefResolver to referencing

[cscetbon]

Trying to follow https://python-jsonschema.readthedocs.io/en/latest/referencing/#migrating-from-refresolver using the latest alpha version but I can't get it to work. I currently have

def get_schema():
  return {
{
    "$schema": "http://json-schema.org/draft-04/schema#",
    "definitions": {
        "edm": {...},
        "eem": {...}
    },
    "anyOf": [
        {
            "$ref": "#/definitions/eem/events/SISAddress"
        },
        {
            "$ref": "#/definitions/eem/events/SISEnrollment"
        },
        {
            "$ref": "#/definitions/eem/events/SISPerson"
        },
        {
            "$ref": "#/definitions/eem/events/SISRestriction"
        },
        {
            "$ref": "#/definitions/eem/events/SISSection"
        }
    ]
}
  }

my_schema = get_schema()
schemastore = {
  'https://my_url/EDM.json': my_schema['definitions']['edm'],
  'https://my_url/EEM.json': my_schema['definitions']['eem']
}

resolver = jsonschema.RefResolver(base_uri='', referrer=my_schema, store=schemastore)
jsonschema.validate(payload, my_schema, resolver=resolver)

I tried to do

my_schema = get_schema()
schema = Resource.from_contents(my_schema)
registry = Registry().with_resources([
  ('https://my_url/EDM.json', my_schema['definitions']['edm']),
  ('https://my_url/EEM.json', my_schema['definitions']['eem'])
])
validator = Draft4Validator(schema=my_schema, registry=registry)
validator.validate(payload)

but I get

...
File ~/.virtualenvs/sds/lib/python3.10/site-packages/referencing/_core.py:394, in Registry.crawl(self)
    391 while uncrawled:
    392     uri, resource = uncrawled.pop()
--> 394     id = resource.id()
    395     if id is not None:
    396         uri = urljoin(uri, id)

AttributeError: 'dict' object has no attribute 'id'

[Julian]

You're passing dicts (object schemas) to the second argument of with_resources, but it takes referencing.Resource objects, which are essentially your dict wrapped with the information that it is a Draft 4 schema.

You could instead either call Resource.from_contents on your dicts if they contain $schema keys, or use referencing.jsonschema.DRAFT4.create_resource(each) on them to explicitly say they're draft 4 resources, or even better would be if they have id keys in them indicating they're subresources (so e.g. "id": "https://my_url/EDM.json" in your definitions/edm schema) in which case you need to do nothing extra at all, they'll be automatically found if your parent schema resource is in the registry.

Hope that helps but if there's suggestions for how you'd have figured the above out in the doc page you linked let me know.

[cscetbon]

Damn I didn't copy the right code 🤦 and I can't touch the definition that already exists except if I patch it manually

In [80]: my_schema = get_schema()
    ...: registry = Registry().with_resources([
    ...:   ('https://my_url/EDM.json', Resource.from_contents(my_schema['definitions']['edm'])),
    ...:   ('https://my_url/EEM.json', Resource.from_contents(my_schema['definitions']['eem']))
    ...: ])
    ...: validator = Draft4Validator(schema=my_schema, registry=registry)
    ...: validator.validate(payload)

...
InvalidAnchor: '#sis/Metadata' is not a valid anchor, neither as a plain name anchor nor as a JSON Pointer.
You may have intended to use '#/sis/Metadata', as the slash is required *before each segment* of a JSON pointer.

But that anchor was working before. Also I'm trying to upgrade in order to overcome #1002, do you think it will make any difference or not a chance ?

[cscetbon]

Not what I'm saying, I was saying that we got errors that are not correctly shown and instead we got the whole schema dumped, similar to what the ticket I pointed shows.

Also I don't get it cause this is how the id is defined

...
        "Metadata": {
            "$id": "#sis/Metadata",
...

So the ref used the id not really an anchor IIUC

[Julian]

I'm not following I'm afraid. And the ID you're showing there isn't valid. Can you show a more realistic example and explain what the question is?

[cscetbon]

I'm afraid the schema is 10 pages long so showing an example is gonna take me some time. Is the schema invalid because of the dashes ? what I can see in the schema is that we have IDs and refs corresponding to those IDs. It also works with https://www.npmjs.com/package/ajv which doesn't complain about it ..

...
                    "$id": "#sis/Metadata",
                    "$id": "#sis/ImplicitType",
                    "$id": "#sis/MetadataWithImplicit",
                    "$id": "#sis/PersonReference",
                    "$id": "#sis/SectionWithSingleCourse",
                    "$id": "#sis/HasStatus",
...

And the corresponding refs

...
                    "$ref": "#sis/Metadata"
                    "$ref": "#sis/PersonReference"
                    "$ref": "#sis/MetadataWithImplicit"
                    "$ref": "#sis/SectionWithSingleCourse"
                    "$ref": "#sis/PersonReference"
                    "$ref": "#sis/HasStatus"
...

Also why is it a problem with the alpha version and not with the current release ?

[Julian]

If those are your IDs I don't need any example, they're invalid!

There are 2 types of fragments in JSON Schema (the specification). (I'll give the examples using Draft 2020 syntax but if you're using some other version it doesn't change the validity of those IDs, just how you declare them) --

• plain name fragments look like #foo, and basically say "go find somewhere that says "$anchor": "foo" and that's the subschema I'm referencing"
• JSON pointer fragments look like #/foo/0/baz which say "go to the root of the current document, then look for a property called foo in it, the first element in that array, and the key called baz in that array element" (in practice this is often #/$defs/something using the JSON Schema $defs keyword)

Neither syntax allows a fragment like #foo/bar -- plain name fragments do not allow slashes at all:

If present, the value of this keyword MUST be a string and MUST start with a letter ([A-Za-z]) or underscore (""), followed by any number of letters, digits ([0-9]), hyphens ("-"), underscores (""), and periods (".").

(from here)

JSON Pointers must start with a slash (see the syntax section of their RFC)

The $id you showed fits neither syntax, it's simply invalid, and furthermore it's ambiguous even if one were to consider it which behavior it should have.

It also works with https://www.npmjs.com/package/ajv which doesn't complain about it ..

Sounds therefore like a bug in ajv! They may appreciate you opening it, though they may not, for the same reason this may have worked in this library previously -- namely, there's a "contract" whenever you're using a library implementing a specification which is basically that the spec is what we're implementing, so you're supposed to stick within things valid under the specification. I'd like to think that jsonschema (now this library) is pretty well tested and pretty rigid about backwards compatibility guarantees -- but that applies only within the "walls" of the spec -- once you go outside it now those things are now no longer true -- there are no tests covering the behavior if you use an ID of #foo/bar in my test suite because that's not a valid ID. It's true the library did something previously, but that's not covered under any backwards compatibility guarantee and I certainly didn't even try out such a thing when improving the $ref support (of course if I thought of it I would have to try to make life easier, but I didn't). If you use some random function (not just in this library) and pass it invalid input, it's not always the case that you get an error, sometimes you get random behavior -- garbage in garbage out. Essentially you can't rely on any behavior there, because to detect every possible invalid input potentially means every user incurs performance cost (of checking) rather than simply "you are expected to give valid input to the function".

Hopefully that's some explanation, but yeah the short answer is again fairly simple, you seem to be correct that the behavior changed for your case, but your schema is not a valid JSON Schema, it contains invalid IDs.

[Julian]

The fix by the way is likely quite simple -- if you mean to use plain name fragments, change the slash to something else, a hyphen or something. If you mean to use JSON pointers, you might want to instead use fragments like foo#bar and add $ids to your subschemas (so "$id": "foo", and now you are indeed allowed to lookup an anchor called bar inside of that).

[cscetbon]

Thanks for the explanation, if I understand correctly (see my first comment) we're using json pointer fragments as refs and in IDs we shouldn't have any # cause it doesn't make any sense there and is not supported. I also don't see any use of $anchor in our schema so the proposed fragment of the form path#anchor_name is not needed.