How should the service handle mandatory fields?

[CharliePatterson]

Context:

• The check tool currently has a concept of ‘mandatory fields’: i.e. a defined list of fields that users must include in any file they run through the check tool. If those fields don’t exist, then the pipeline returns an error to tell the user that these don’t exist in the file and that they need to include them,
• This list is currently hard-coded, and is one of the things that prevents the service from being fully dynamic, and being able to automatically add new specifications.

Questions to answer:

1. Should the service continue to have a concept of mandatory fields?
2. If yes, then how should these be handled, to ensure that the service can be fully dynamic and automatically integrate new specifications?

Meeting summary:

The following notes are from a meeting between folks from the Data Design, Data Management, Infrastructure and Providers teams:

• It was proposed that the concept of mandatory fields should be abolished (bar the reference field, which should be included as a bare minimum):
  • Any fields that are listed in the specification are ones that the LPA should provide.
  • However, we want to encourage LPAs to submit some data, and iteratively improve it, rather than waiting till they’ve assembled a complete dataset with all required fields.
  • As such, it’s perfectly legitimate for an LPA to submit a file that simply includes a list of reference fields.
• Brownfield land is an exception to this, since the legislation describes/lists out each of the fields (NB: this isn’t going to be the case for any future datasets, as the legislation will just point at the specification, which can evolve over time). In this instance, we will continue to have a set of mandatory fields (outlined here).
• The Providers team should iterate on the design of the check tool, so that it’s clear to users that they can publish and submit data that doesn’t include all the fields outlined in the specification.
• Alongside that, we should also unwind the concept of mandatory fields for all datasets, with the exception of brownfield land, and the exception of the reference field for all datasets. This should be a joint effort between Providers and Infrastructure.
• There’s an open question of how the service should handle missing / blank entries: the specifications don’t have a concept of this currently.

Actions:

1. Providers team to do design and research work to look at how to iterate the check tool (on the roadmap for this quarter) in line with the above.
2. Alongside this, we will need to collaborate on changing the approach for mandatory fields.
3. Separate discussion required to iron out how we handle missing / blank entries.