"Flat" vs "structured" Murmurations fields

[Photosynthesis]

What this is about

Some of the fields in the Murmurations library can be added as a single field in a schema, but include multiple "child" fields or sub-fields within the field definition. Examples of this include the location field (which includes country, region, and locality as sub-fields), and the geolocation field (which includes lat and lon subfields). Fields that include sub-fields in their definition I'm calling "structured" fields, because the data that matches that field has internal structure to it. What I’m calling “flat” fields, on the other hand, do not have internal structure. One way to think about this is that data that matches flat fields can be put into a straightforward table, unlike structured fields.

Example structured JSON data:

{
    "geolocation" : {
        "lat" : 123,
        "lon" : 42
    }
    "location" : {
        "country" : "GB",
        "region" : "England",
        "locality" : "London"
    }
}

Example equivalent flat JSON data:

{
    "lat" : 123
    "lon" : 42
    "country" : "GB",
    "region" : "England",
    "locality" : "London"
}

Issues with structured fields

Lack of flexibility for schemas

If a schema creator wants to include either:

• Just some of the components of the location field (for example country only); or
• A different format of any of the components of the location field (for example a non ISO 3166 country); or
• An additional location-related field (eg, postal code)

The location field as defined won't suffice. An example of this is the KVM data, which has non-ISO country names, making the existing location field unusable.

Difficulty handling in aggregators

For most aggregators, it will be a lot easier to handle flat data than structured data. This is because in most cases data will be stored in tables, not as JSON documents. For array fields, structure is necessary. But for fields like location and geolocation, there doesn't seem to be any functional advantage to put these fields into a structure.

Can't pass schema field to RJSF for filters

Case in point: a common need for an aggregator would be to filter nodes based on some component of the location (often country). The convenient way to do this is to allow aggregator operators to choose fields from the schema, which then get passed to an RJSF form for client-side filters. But, this won't work if location data is grouped into a structured field.

Need pre-processing to enable location component search/queries

On the back-end, if an aggregator wants to allow querying by location components (which is a common need), they will generally have to do additional work to break apart the location data into flat data and store it separately.

Problematic for input/editing forms

Flat data is easier to handle for forms. Even in cases where we can use RJSF, making the form look good gets much more annoying when you have structured fields, because RJSF does a very poor job of adding convenient CSS selectors to fields, so you have to (at least in my experience) include some of the structural aspects in your selector to get at a specific input. When you're dealing with data of an unpredictable shape (as in any general purpose aggregator), this becomes problematic.

Advantages of structured fields

• Keeps logically related data together
• Follows more the pattern of linked data and schema.org (though existing structured Murmurations fields don’t seem to match any schema.org specs)
• Makes use of JSON’s ability to structure data
• Can allow disambiguating fields with the same or similar names and content but used for different purposes

Proposed change

So, my suggestion is:

• Create country, region, locality, latitude, and longitude fields in the library
• Update the location and geolocation fields to ref the new library fields (rather than having the subfields defined inside the parent field). This means that existing structured fields remain backwards compatible, without duplicate field definitions.
• Update the index to work with flat location data (in addition to structured)
• Encourage schema creators to use flat fields unless there is a specific need for structured fields

[olisb]

like it - the proposed changes make sense to me. +1