Improvements on getForms.js to remove anti-patterns

[mtuchi]

Background

The getForms.js job contain couple of anti-patterns that we can avoid by flattening the operations and break down the job into two jobs and leverage the edge condition feature on lightning. Below are couple of improvements that can simplify the job

1st - Create new input when using manualCursor

In lightning you can manually create a initial state by creating new input for any step in your workflow. This is great for specifying manual cursors for your workflow or any initial input that you want to test in your workflow. For getForms.js we have two manual cursor [cursorStart and cursorEnd] both can be specified as initial input instead of hard coding them in the job.

An example for setting up the cursor using initial state

state.json

{
    "startOfMonthPrior": "2021-01-01",
    "endOfMonthPrior": "2021-01-31"
}

getForms.js

fn(state => {
  const today = new Date();
  const month = today.getMonth();
  const year = today.getFullYear();
  const { endOfMonthPrior, startOfMonthPrior } = state;

  state.cursorStart =
    startOfMonthPrior ??
    new Date(year, month - 1, 1).toISOString().split('T')[0];

  state.cursorEnd =
    endOfMonthPrior ?? new Date(year, month, 0).toISOString().split('T')[0];

  console.log('Cursor end:', state.cursorEnd);
  console.log('Cursor start:', state.cursorStart);

  return state;
});

// Other operations below

2nd - Avoid nested operations

Currently when fetching form list from commcare we use the callback to post the list of filtered forms to openfn inbox (Another webhook workflow in v2). This introduces a lot of ant-patterns because we have nested operations that needs to be called with state. To avoid this, we can remove the each() operation from the callback and create a new down stream job that will use the final state of getForms.js. The down stream job (We can call it postCleanedForms.js) should be triggered only if we have a list of cleaned forms

Flattened getForms.js job

㊙️ state.json

{
 "baseUrl": "https://www.commcarehq.org/a/miraclefeet/api/v0.5",
 "password": "LP",
 "username": "LP",
}

👨🏽‍💻 getForms.js | [email protected]

const setOffSet = meta => (meta?.next ? meta.limit + meta.offset : 0);

fn(state => {
  const today = new Date();
  const month = today.getMonth();
  const year = today.getFullYear();
  const { endOfMonthPrior, startOfMonthPrior } = state;

  state.cursorStart =
    startOfMonthPrior ??
    new Date(year, month - 1, 1).toISOString().split('T')[0];

  state.cursorEnd =
    endOfMonthPrior ?? new Date(year, month, 0).toISOString().split('T')[0];

  console.log('Cursor end:', state.cursorEnd);
  console.log('Cursor start:', state.cursorStart);

  return state;
});

get(
  '/form/',
  {
    query: {
      limit: 200, // To update for fetching a specific number of forms
      offset: setOffSet($.meta),
      xmlns:
        'http://openrosa.org/formdesigner/34532e2a938003aae150198e2613ed2b54c410fb', // Register New Patient
      received_on_start: $.cursorStart,
      received_on_end: $.cursorEnd,
    },
  },
  state => {
    const { meta, objects } = state.data;
    state.meta = meta;
    console.log('Metadata in CommCare response for Register New Patient:');
    console.log(meta);

    state.cleanedForms = objects.filter(form => {
      const { received_on, server_modified_on } = form;
      const receivedOnDate = new Date(received_on);
      const serverModifiedOnDate = new Date(server_modified_on);
      receivedOnDate.setHours(
        receivedOnDate.getHours(),
        receivedOnDate.getMinutes(),
        0,
        0
      );
      serverModifiedOnDate.setHours(
        serverModifiedOnDate.getHours(),
        serverModifiedOnDate.getMinutes(),
        0,
        0
      );
      return (
        receivedOnDate.toISOString() !== serverModifiedOnDate.toISOString()
      );
    });

    if (state.cleanedForms.length > 0) {
      console.log(state.cleanedForms.length, 'cleaned forms fetched...');
      console.log(
        `Posting to OpenFn Inbox...${JSON.stringify(
          state.cleanedForms,
          null,
          2
        )}`
      );
    }

    return state;
  }
);

Edge condition state.cleanedForms.length > 0

postCleanedForms.js

🌵 state.json

{
    "baseUrl": "OPENF_WEBHOOK_URL",
    "password": "webhook-password",
    "username": "webhook-username"
}

postCleanedForms.js | [email protected]

each(
  $.cleanedForms,
  post(
    '/',
    {
      body: $.data,
      headers: { 'content-type': 'application/json' },
    },
    ({ data, references, ...remaingState }) => remaingState
  )
);

PS: You might have noticed the use of $ operator in my examples
That is a new feature called Lazy State Operator, To learn more see the documentation here

workflow.json for getForms and postCleanedForms steps

{
    "options": {
        "start": "getForms"
    },
    "workflow": {
        "steps": [
            {
                "id": "getForms",
                "adaptor": "http",
                "configuration": "commcare-creds.json",
                "expression": "../getForms.js",
                "state": {
                    "startOfMonthPrior": "2021-01-01",
                    "endOfMonthPrior": "2021-01-31"
                },
                "next": {
                    "postCleanedForms": "state.cleanedForms.length > 0"
                }
            },
            {
                "id": "postCleanedForms",
                "adaptor": "http",
                "configuration": "openfn-creds.json",
                "expression": "../postCleanedForms.js"
            }
        ]
    }
}

[ritazagoni]

hey @mtuchi on the 1st: we'd like to have the dynamic previous month as default cursor. I just change the order of expression to evaluate the below and the state.cursorEnd, right?

  state.cursorStart =
    startOfMonthPrior ??
    new Date(year, month - 1, 1).toISOString().split('T')[0];

[mtuchi]

@ritazagoni if you want to use the default cursor aka new Date(year, month - 1, 1).toISoString().split('T')[0]. Then you don't need to specify state. startOfMonthPrior in your initial state.

You can simply create an input with only endOfMonth Eg

 { "endOfMonthPrior": "2021-01-31" }