Enhancement request: provide specific documentation on how to make schema validation use only local files

[rohan0rtiz0]

I want to make Schema validation use only local files on the filesystem, even if the "$id" field has an HTTP URL. (So that my script doesn't continuously hit the website.) I can't figure out how to do that based on the current documentation. My request is that you include documentation about how to perform this specific type of usage.

(Note: local-file-only validation works for me with check-jsonschema --base-uri ., but because the check-jsonschema code is too object-oriented and complicated, I can't figure out how to do the same thing with just.

Also note: ChatGPT has absorbed only the older RefResolver approach, which gives a deprecation warning. (Probably from places like StackOverflow where old reference to that from 2019 are the top hit.). And it's not able to figure out / help me with doing this the right way either.)

[Julian]

Hi there -- within this library, or as far as referencing.Registry is concerned, nothing will make an HTTP request unless you yourself have done so, so there's not much to document. It's possible you're using something downstream of this library where you're making HTTP requests, e.g. the jsonschema library will do so for backwards compatibility reasons until version 5 is released, but will emit a warning if you rely on this behavior (and it sounds like you're using check-jsonschema, so you may want to consult there instead.

[rohan0rtiz0]

Hello and thanks for the quick reply. I was specifically using jsonschema validate() in my own python script. I was concerned that if I filed this ticket against jsonschema it would be said that the correct solution is to use referencing since that's what the deprecation warning says. I only mention check-jsonschema because it's an example of having something that documents how to stop network lookup. Should this instead be re-filed as a documentation request against jsonschema?

[Julian]

Should this instead be re-filed as a documentation request against jsonschema?

(I've moved this there, so no worries on re-filing.)

[Julian]

Got it, then the docs you should have a look at are here -- specifically as long as you provide your own referencing.Registry, no HTTP requests are made -- that's specifically called out at the bottom here.

[rohan0rtiz0]

Thanks, but it's still not clear to me how to translate that example into a "just use local files for lookup" example. I tried the following in the spirit of --base-uri .

    with open(in_filename, 'r') as f:
        instance = json.load(f)

    with open("MySchema.json", 'r') as f:
        MySchema = json.load(f)

    schema = Resource(MySchema, specification=DRAFT202012)
    registry = Registry().with_resource(uri=".", resource=schema)
    print(registry.contents(".")) # This code works, but prints only the single file I imported, not further referenced files. I don't want to import all files, I want them to just be resolved as all local files automatically from the starter schema, like with check-jsonschema.

    for entry in instance:
        try:
            validate(instance=entry, schema=MySchema, registry=registry)
<cut>

This gives an error:

Traceback (most recent call last):
  File "/home/user/.local/lib/python3.8/site-packages/jsonschema/validators.py", line 463, in _validate_reference
    resolved = self._resolver.lookup(ref)
  File "/home/user/.local/lib/python3.8/site-packages/referencing/_core.py", line 674, in lookup
    raise exceptions.Unresolvable(ref=ref) from None
referencing.exceptions.Unresolvable: File2.json#/properties/ThingArray

Where File2.json is in the same directory as MySchema.json, and referenced from within it as

"ThingArray": { "$ref": "File2.json#/properties/ThingArray" }

(And again, I know the schema and referencing is all fine right now, because both online lookup works, and check-jsonschema works. So it's just how to force it to use local files only which I'm not clear on.)

[Julian]

. is not the URI of your schema, the URI would include whatever file path it has. But it sounds like you're asking precisely how to do https://python-jsonschema.readthedocs.io/en/stable/referencing/#resolving-references-from-the-file-system which recommends loading all your schemas in that directory, or if you really don't want to, give a way to do so lazily.

[rohan0rtiz0]

That looks somewhat like what I'm trying to do, but will that http://localhost approach work with a relative URL of ./ or only with absolute URLs like "/tmp/foo"? Because it wouldn't work to encode hardcoded absolute paths because people can check out this script anywhere, and I want the script to reference the git repo it's contained within, not a /tmp path)

(I just changed it to

SCHEMAS = Path("./")
<provided code for retrieve_from_filesystem()>
registry = Registry(retrieve=retrieve_from_filesystem)

But it still gets the same referencing.exceptions.Unresolvable: File2.json#/properties/ThingArray error.)

Likewise, I don't want to import all files up front, because the Schema is under development with many referenced sub-files popping in and out of existence as the design gets figured out. So it's best if just the base schema can be provided and then it gets resolved in the same way as network lookups, just via the local filesystem.

[Julian]

/tmp/foo is not an absolute URI, it's a path. The absolute URI it corresponds to is file:///tmp/foo.

Whether you use absolute URIs or relative URIs, and whether you use file URIs or other URIs for your schemas, are two somewhat orthogonal choices, and all somewhat orthogonal to how you load these schemas into a registry.

In brief, my recommendation is to use absolute URIs everywhere, and to use tag: or urn: URIs rather than file URIs entirely, exactly for the reason you mention which is that paths differ across machines. But you can certainly use http or file URIs and reference things via relative URIs, but then you need to provide a correct base URI when creating your registry.

Likewise, I don't want to import all files up front, because the Schema is under development with many referenced sub-files popping in and out of existence as the design gets figured out.

I'm not sure I understand the relevance of this, unless you mean that you're going to start up a long running program and then change stuff while it's running and you want those changes to be reflected in the live program immediately (in which case yes, that's a bit of what the retrieval logic I linked is meant to solve).

[rohan0rtiz0]

/tmp/foo is not an absolute URI, it's a path. The absolute URI it corresponds to is file:///tmp/foo

I meant to say path not URL, but thanks for clarifying.

But you can certainly use http or file URIs and reference things via relative URIs, but then you need to provide a correct base URI when creating your registry.

And what specific "correct base URI" is that for the ./ directory? That's what I'm asking to have documented in an example. I've got absolute http:// URIs for the "$id" fields, but things like check-jsonschema can seemingly simply override it from "https://foo.com/dir/File2.json" to "./File2.json", and that's what I'm still requesting you to explain how to do for validate(). (Perhaps you're thinking I'm more familiar with JSON schemas than I am. And yes, I know that the docs say you're supposed to be familiar with JSON schemas already. And I am, in the sense that I just started on this and just got things to the point where checked-jsonschema validates everything correctly. Now I just need to incorporate validation into my own script instead of using a CLI tool.)

I'm not sure I understand the relevance of this

The user-friendliness of check-jsonschema with --base-uri . is that nothing needs to be changed when a sub-file referenced by a base schema is changed. One always just points at the base schema, and check-jsonschema figures out for itself where the other local files are that are referenced by the base. That's the change-minimization I'm trying to replicate here. If I had to keep changing the code each time a new file was created or removed, that wouldn't be as user friendly as just pointing at the single base file and letting the system resolve where the other files are from the . directory.

[Julian]

And what specific "correct base URI" is that for the ./ directory?

The only one it implicitly might have is file:///path/to/your/directory -- that of course is a poor one to use when sharing the schema around, which is why as I say, usually you should assign it a URI you want to use, and then reference schemas using the URIs you're using. Again, absolute ones are often easiest here as they are immune to renaming even, as is just loading all the schemas off the filesystem ahead of time.

If I had to keep changing the code each time a new file was created or removed, that wouldn't be as user friendly as just pointing at the single base file and letting the system resolve where the other files are from the . directory.

This of course isn't necessary! You are writing a program, you can do whatever dynamic things you want, including what's intended here for this, which is "walk your directory with e.g. pathlib.Path(".").walk or .rglob and simply put all those in a registry -- doing so is two lines of code, e.g. (unverified, so treat it as pseudocode) registry = [referencing.Resource.from_contents(json.loads(path.read_text())) for path in YOUR_SCHEMA_DIR.rglob("*.json")] @ referencing.Registry() under the assumption that all your schemas properly declare their $schema and $id keywords, that you put some reasonable URIs you want to use in them (like the tag: URIs or urn: URIs or http:// URIs) and that your $ref keywords all use these URIs.

Yes I sympathize that a decent portion of this is general JSON Schema (or really general URI!) knowledge, not specifically related to referencing which is simply ready for "ok now you tell me what URIs you want to assign". I'm afraid I don't have time to write something out in more detail than what's already there at the moment, though I do think if you went through the docs slowly that the answers are there, even if they're not simply 30 seconds to find -- I'm afraid URIs, $ref and generally schema resolution are simply not easy concepts until they click in your head.