Store a history of the process count

[duncandewhurst]

When a data source has collection errors it's difficult to tell how much data is missing, for example, the most recent Kyrgyzstan dataset reports one error and has ~144k processes. Previously, datasets for Kyrgyzstan had ~675k processes.

As an analyst, I would like to see the history of the process count for each dataset so that I can quickly identify if there is a significant amount of missing data.

cc @mrshll1001

[jpmckinney]

Hi @duncandewhurst, can you please document your use of this project in terms of publisher feedback?

cc @yolile

[duncandewhurst]

I don't use it for publisher feedback. I use it for ad-hoc queries and for data use training as discussed in open-contracting/ocdskit#75 (comment), CRM-7149, https://opencontractingint.slack.com/archives/C01N5FPUWNM/p1626272242005000 and https://opencontractingint.slack.com/archives/C01N5FPUWNM/p1633382613005800.

[jpmckinney]

👍 Since the GitHub issue was about a quality issue, I thought it might be related to publisher feedback.

[duncandewhurst]

Just something that I noticed whilst preparing for a training :-)

[kindly]

@duncandewhurst sorry missed this.

We actually keep not just the process count but the count for every field for every scrape. We just do not expose that to the user yet.