Mutable registry and schema references in reified schemas

[DerGuteMoritz]

Hi,

I'm having an issue with the combination of mutable registries and schema references, namely that references to schemas in the registry are dereferenced only once when a schema is reified. This means that any further updates to the referenced schema in the registry are not seen by the referencing schema. I wonder if there already is a way to make this a bit more dynamic, e.g. similar to how Clojure itself has var-quote or the :redef annotation for dynamically resolving vars even when directly linking. If not, some guidance around how to approach this would be greatly appreciated 🙂

---

To illustrate what I mean, here's an example scenario. This is assuming the process was started with -Dmalli.registry/type=custom, of course. Here's a namespace that provides access to the mutable default registry:

(ns example.schema
  (:require [malli.core :as m]
            [malli.registry :as mr]))

(defonce registry
  (atom (m/default-schemas)))

(defn register [schema-key schema]
  (swap! registry assoc schema-key schema))

(mr/set-default-registry!
 (mr/mutable-registry registry))

And here's two namespaces making use of it:

(ns example.foo
  (:require [example.schema :as schema]))

(schema/register ::bar :string)


(ns example.bar
  (:require [example.foo :as foo]
            [example.schema :as schema]))

(schema/register ::qux [:map ::foo/bar])

Now something like this works fine:

(m/validate :example.bar/qux {:example.foo/bar "hello"}) ; => true

And since we never reified the schema before using it, any changes to :example.foo/bar would also be reflected in :example.bar/qux. However, once we start doing that, e.g. by applying some schema transformations, this is no longer the case. Say we first change example.bar like this:

(ns example.bar
  (:require [example.foo :as foo]
            [example.schema :as schema]
            [malli.util :as mu]))

(schema/register ::qux (mu/closed-schema [:map ::foo/bar]))

Now ::qux is stored as a reified schema in the registry. If we now
change :example.foo/bar to also allow an :int like this:

(ns example.foo
  (:require [example.schema :as schema]))

(schema/register ::bar [:or :string :int])

That change is not picked up by :example.bar/qux anymore:

(m/explain :example.bar/qux {:example.foo/bar 123})

; =>

{:schema :example.bar/qux,
 :value #:example.foo{:bar 123},
 :errors
 ({:path [:example.foo/bar],
   :in [:example.foo/bar],
   :schema :string,
   :value 123,
   :type nil,
   :message nil})}

I am aware of two ways to deal with this issue:

1. Recompile example.bar whenever example.foo changes.
2. Always run schemas through m/form when registering them.

However, both of them are quite coarse grained and lead to a lot of noticeable overhead in practice once the reference chains get longer. Also, option 1 only really works for interactive use.

As mentioned above, ideally I would like references to work a bit more like redefable vars (maybe via an option), so that even when the referencing schema is reified, the reference would be checked against the current value of the registry on use and re-reified if necessary. Hope that make sense!

[nilern]

1. Is what the Self (and HotSpot) JIT does (section 6). There is hardly any overhead except when things actually do change.

[DerGuteMoritz]

Wow, thanks a lot, that was really helpful! Particularly this bit at the beginning of section 6.1 (emphasis mine):

A high-productivity programming environment requires that programming changes take effect within a fraction of a second. This is accomplished in our SELF system by selectively invalidating only those compiled methods that are affected by the programming change, recompiling them from new definitions when next needed. The compiler maintains two-way change dependency links between each cached compiled method and the slots that the compiled method depends on.

I went ahead and tried to implement that approach. Turns out that it neatly solves the problem, indeed! A nice side-effect of it is that I can now always store reified schemas in the registry without regret which improves performance throughout (e.g. for instantiating coders etc.). Since the code got a bit longer, I pasted it over here: https://gist.github.com/DerGuteMoritz/82718d85cc70c983c75a56e77786d4db -- not 100% sure about everything in there, especially schema-references and dynamic-ref-schema. Feedback very welcome.

I guess this kind of thing is out of scope for malli proper? If so, I will look into maybe rolling it into a utility library.

[nilern]

That's great! And here I thought I was just bragging about having read that JIT stuff.

Malli does have a general tension where on one hand we try to precompute as much as possible (in e.g. validator) but on the other hand want to support open world scenarios with mutable registries, schema serialization etc. From that perspective this could be an empowering addition to Malli. But for things to Just Work, all the generated validator etc. functions would need to be invalidated as well and that might be too much, especially if any responsibility for that leaks to custom schema protocol implementations.

In any case, make sure that your code does not loop forever on recursive schemas.

[DerGuteMoritz]

That's great! And here I thought I was just bragging about having read that JIT stuff.

😂

Malli does have a general tension where on one hand we try to precompute as much as possible (in e.g. validator) but on the other hand want to support open world scenarios with mutable registries, schema serialization etc. From that perspective this could be an empowering addition to Malli.

True, I've got a pretty good taste of that tension by now 😄 Let's see where this endeavor leads... I'd certainly be happy to contribute this functionality to be part of Malli.

But for things to Just Work, all the generated validator etc. functions would need to be invalidated as well and that might be too much, especially if any responsibility for that leaks to custom schema protocol implementations.

Agreed. I have a rough idea how the appraoch could be extended to validators etc. as well. So far the lack of this didn't cause any trouble for us but I guess that's just a matter of time. I'll try to cook something up soon ™️

In any case, make sure that your code does not loop forever on recursive schemas.

Good catch! Initially I didn't care for this case because I thought it would be impossible to actually declare recursive references because the recursive registry lookup would fail during reification in register. But if one registers a non-recursive version of the schema before the recursive one, that's of course perfectly possible. Updated the gist to handle this properly.

[nilern]

So far the lack of this didn't cause any trouble for us but I guess that's just a matter of time.

If e.g. one only uses Malli for Reitit validation and has a reloaded workflow where the router is regenerated on every reload trouble is unlikely. But often one would like to do more with Malli or has a different router etc.

Good catch!

Recursion always causes trouble in language implementation, I've learned in great detail.