fetchOSD upgrades

[brownag]

Merging functionality w/ @dylanbeaudette (https://github.com/ncss-tech/parse-osd/) for weekly rebuilds of OSD-related data snapshots via https://github.com/ncss-tech/OSDRegistry and https://github.com/ncss-tech/SoilKnowledgeBase

• Move get_OSD(url, what = c("txt","html")) routines from parse-osd repo to {soilDB}

  • should work with official URL, custom URL/mirror, and local files; useful for checking text-based products off cycle from SKB refresh

• Re-factor downloadParseSave() code to use OSD JSON files produced by {SoilKnowledgeBase}

  • New {SoilKnowledgeBase} OSD products to be created by SKB create_OSD.R in OSD folder:
    • parsed site and horizon data tables are included in main JSON result

• Set up get_OSD() as interface for JSON, HTML or TXT sources of OSD data

• Incorporate JSON data sources into fetchOSD()

• need to be able to splice in "inferred" data (such as colors predicted by OLS) for parity with old result

For further discussion on source data used by OSD_query() and other OSD-derived products see ncss-tech/SoilKnowledgeBase#25

[dylanbeaudette]

See preliminary re-factored code in R/OSD-file-access.R. Code doesn't have to stay here, just a convenient parking place.

[brownag]

Consolidated the OSD file access into a new method get_OSD() which works on local or remote JSON, HTML and TXT sources of OSDs.

The following items are outside the scope of soilDB-specific updates so they have been removed from the task list for this issue.

• table/identifiers of missing colors predicted via OLS(value, chroma)
• "complete" full-text .sql file for full-text search
• "sectionized" full-text .sql file for full-text search

The OSD JSON parsing method and new structure is robust enough to add new data at site or horizon level and associated identifiers as needed. The .sql files for full-text search are used only for SoilWeb backend of OSD_query() so the methods used to generate those files do not need to be addressed in this issue.

[dylanbeaudette]

• need to be able to splice in "inferred" data (such as colors predicted by OLS) for parity with old result

This was added to the pre-processing code, SoilWeb API, and fetchOSD() code some time back.

[dylanbeaudette]

• Incorporate JSON data sources into fetchOSD()

Now that we have get_OSD, I think that we can safely make this item as "done", and close the issue--yeah?

[brownag]

• Incorporate JSON data sources into fetchOSD()

Now that we have get_OSD, I think that we can safely make this item as "done", and close the issue--yeah?

Yes agreed. It is as integrated as it is going to be from perspective of soilDB.