Incremental Loading - Stage 1 - Implement checks (3) - Config changed

[Ben-Hodgkiss]

Overview
We need to check if the configuration files have changed since the last successful run. To do this we need to find the last changes to them. While it is possible to use he github API to find the last checkin to the repo, it may well be simpler and clearer to update the process to do a lightweight checkout. The git log can then be checked for the relevant files.

Tech Approach

• Update makerules to use git to checkout the config dirs, rather than getting the files via the raw.github URL.
• Determine the files used for this collection.
• Use git log to find the last update of the relevant files
• Use this information, along with the last successful run, to add to the process log to determine this collection should be processed.

Acceptance Criteria/Tests

• The collection runs as normal despite the changes to how the files are obtained.
• Changes to the relevant parts of the config are determined
• The process log is correctly updates

Resourcing & Dependencies

• Developer, will need docker setup to test the container changes.