Update fetch-metadata-and-generate-opts-json workflow to also fetch and append omrs-forms-metadata and identifiers sheet to the metadata-json

[AishaHassen]

Background, context, and business value

New sheet has been added to the metadata file to ensure that we are not hardcoding list of sheets to fetch and their corresponding omrs-form-uuid, dhis2-program, dhis2-program-stage. The ask of this issue is to get those sheets and add them to the metadata.json file being saved on Github.

The specific request, in as few words as possible

1. Modify the Get metadata file from Sharepoint Job

• First fetch only the form-metadata sheet.
• Using that sheet construct a JSON named state.form-metadata as shown below:

​​[
    {
        "OMRS form name": "F01-MHPSS Baseline",
        "OMRS form.uuid": "6a3e1e0e-dd13-3465-b8f5-ee2d42691fe5",
        "DHIS2 program ID": "w9MSPn5oSqp",
        "DHIS2 programStage ID": "MdTtRixaC1B",
        "OMRS Form Version": "v1"
    },
    {
        "OMRS form name": "F03-mhGAP Baseline",
        "OMRS form.uuid": "82db23a1-4eb1-3f3c-bb65-b7ebfe95b19b",
        "DHIS2 program ID": "w9MSPn5oSqp",
        "DHIS2 programStage ID": "EZJ9FsNau7Q",
        "OMRS Form Version": "v1"
    }]

• Loop over the above json array to fetch the sheet for the OMRS forms(F01-MHPSS Baseline, F03-mhGAP Baseline) as opposed to looping a hardcoded list in the job code.
• Instead of putting the output json directly on the state object, compile them under state.form-mappings so we can dynamically reference them in the subsequent steps.
• Next fetch a sheet named identifiers, construct and JSON array from the table. and save it under state.identifiers
• This is what the state object should look like after this step:

{
  "form-mappings": {
    "f01MhpssBaseline": [],
    "f02MhpssFollowUp": [],
     .....
  },
  "forms-metadata": [],
  "identifiers": [],
  "optionsets": []
}

2. Modify the Save option-set json to github step

• In the last step that merges the JSON objects into one large JSON Object, extract the contents of state.form-mappings and include state.form-metadata into the final JSON Object.

state.json

{}

Name of File to fetch from Sharepoing

LIME EMR - Iraq Metadata - Release 1

[aleksa-krolls]

@mtuchi as discussed on the call:

1. Step 2 wf/2-mappings.js becomes getMetadataCollection to go get the metadata.json to reference in the rest of the job
2. Let's generate this formsMap in WF3 to add to the collection metadata.json
3. Then get these formuids wf2/3-get-encounters.js
4. And then update wf2/4-get-options-map.js to get this formsMap from the collection

And then let's make the TEI mappings more dynamic:

1. Job wf2/2-mappings.js should pull options from the OptionSets sheet instead of hardcoding the mappings
2. For the mappings that reference a hardcoded attribute ID (e.g., nationality, current status, etc.) --> You can using the mappings in F00-Registration form to try to dynamically produce the same output as wf2/2-upsert-teis.js
3. Hopefully we can dynamically set these identifier variables by pulling these identifierType IDs from the identifiers sheet. (Some attributes and identifiers will require custom mappings and data cleaning functions... but at least these ID variables should be set based on the metadata.)

[mtuchi]

EOD Update Work in progress #60

I have manage to complete the mappings and pushing the new generated metadata to metadata/collection.json in next-staging branch to avoid conflicts on staging. And also i have started updating wf2. I need to run the wf2 couple of times to validate my logic for the new mappings before passing the PR to code review. I will continue on this tomorrow

[mtuchi]

The implementation for this issue is done, I have handed over the #60 to Joe for code-review

[aleksa-krolls]

@mtuchi after wrapping up to NH, pls prioritize implementing code review feedback here

[mtuchi]

EOD Update #60

I have update the mapping in wf2 and improve the auto generation of collection.json in wf3. The PR is ready for code review, Once Joe approve i will merge to sync the changes to msf-lime-test project
cc @aleksa-krolls @AishaHassen