Why is merging commutative?

[asymmetric]

Having used Jsonnet for a bit, I was quite surprised to learn that in Nickel, one can't do something like

{ networking.open_ports = [ 123 ] } & { networking.open_ports = [ 456 ] }

expecting the result to be { networking.open_ports = [ 123, 456 ] }.

I assume this is because the & operator in Nickel is commutative, and arrays are ordered. In Jsonnet OTOH, the merge operator (+) gives precedence to the RHS.

Why does Nickel not behave this way? Why is symmetry important?

Why can't the & operator subsume the @ operator for arrays?

[yannham]

Symmetry was taken from both CUE and the NixOS module system approach. The motivation in CUE is discussed here for example, and in other places of the documentation. For NixOS, the user is not the one doing the merge, but it's the NixOS module library. In that context, you don't control the order of merging, it is unspecified, so you need to rely on something else to control which value takes precedence over the other.

But your case has actually two aspects. if & doesn't act on arrays as @, it is first and foremost because there seem to be no universal and natural definition of merging on arrays. You could also decide that [x, y] & [a,b] is [x & a, y & b] for example (that would actually be more consistent with merging for records). You could fail if the size are different, or just take the additional values of the longest array, etc.

However, as you say, merging other things that records is still very valuable in practice. This is why RFC001 proposes to let the user specify custom merge functions. You could then write your example as:

{ networking.open_ports | merge (@) = [ 123 ] } & { networking.open_ports = [ 456 ] }

(or something like that, leaving a few details out). Custom merge functions ought to be commutative too, but this notion has to be nuanced. Here, we use an array for open ports, but in fact the order is not important: actually, the right data structure we are thinking of is an unordered set, not an array. But arrays are a good enough representation. In that situation, while @ is not strictly commutative, it's fine, as long as the final system consuming the configuration doesn't make a difference between |123, 456] and [456, 123]. If you want to be strict, you could even keep the array ordered, making the merge function technically commutative.

Sorry for the long post, so let's recap:

• & is only defined where it has a unique natural definition, which is mostly for records. Otherwise, you'll be able to use custom merge functions, although those are not yet implemented.
• merging and merging functions should be commutative, but in a relaxed sense, i.e. "in spirit", as long as the final configuration is equivalent for the system consuming it. If you need non-commutative operations, you will also be able to use priorities to order things (let's say you concatenate shell scripts: you can put low priority scripts before high priority scripts). Merge priorities are precisely made to distinguish between merge operands without relying on order.

RFC001 has been written some time ago, and hasn't been a priority, but we started implementing it recently (see e.g. #829). I think custom merge functions are not too far in the pipeline 🙂

[majidaldo]

ya, found this conversation wondering why not be able to merge scripts. i'd need to ya either define a custom merging or do it myself before giving it to the consuming system [script1, script2, ...] => script.