Curate CONUS404

[akrherz]

Some initial reviews of ERA5Land curated via #321, have gone up in horrible flames with a soil temperature/moisture product that seems to have major issues over Iowa :( :( :( So, I am going to try CONUS404 and prepare to be disappointed.

[akrherz]

So the initial pain is to resolve what the 4 soil levels are in this dataset. I have a helpdesk ticket into NCAR RDA on this topic.

[akrherz]

RDA support fixed the website documentation and pointed me toward the wrfconstants.nc. The soil layers are

• 0-10 cm
• 10-40 cm
• 40-100 cm
• 100-200 cm

[akrherz]

Replicating the ERA5Land variables, we come up with 256GB per year netcdf files. Life choices need to be made here.

[akrherz]

Thinking aloud about why I am even attempting to curate this dataset...

I am at a combined 15 variables + levels, so any removals would only be saving ~12 GB per. I have space for 256 GB per year, which is about ~10 TB total.

1. I really want something to produce a soil moisture/temperature climatology, so to QC the ISUSM network and to generate plots to answer researcher/public questions. Hourly data for the deep layers makes little sense, but likely would be of value for shallow depths (+8).
2. Evaporation is on the present list, but I likely have no immediate usage for it.
3. Wind data as well, I don't really have an immediate need for this.
4. Precipitation data likely has downstream use and should stay (+1).
5. Solar radiation should be useful (+1)
6. Temperature and Dew Point are likely needed, but again could perhaps just be on a daily time step.

So by trimming, I would save about 80 GB per year. Is that enough to fuss over? Erm. So I am likely to:

• curate the hourly dataset and place on an unshared ZFS storage for a processing machine
• create climatology products afterwards that are shared over mesonet2 fileshare.

[akrherz]

The processing script is currently up to 1998. I should run a quick diagnostic to ensure that I am not getting my hopes up for nuttin!

[akrherz]

We need to spitball what to compute for a soil moisture / temperature climatology, so to be able to place a current value into some historical context.

The first, and most eminent issue, is that we need to put context on the reported soil moisture value so that we are comparing apples and apples. For example, a given sensor may have observed a moisture range (wilting and saturation) between 0.09 and 0.40 and the model has a range between 0.19 and 0.44. So the two values are not directly comparable.

• Create a 42 year min and max value and hope that this time span experienced a wilting and saturation condition!

Having percentiles would be very nice, but the question is storage/speed. If we did this at a weekly time step, with ~40 percentile thresholds on the native grid, that's 12 billion data points. I need to look at some actual maps of data and see what type of variability is found in this model...

[akrherz]

Up to 2012 now, but an initial plot of soil moisture climatology was a debbie downer.

[akrherz]

Dataset fully downloaded now (thru 2021). Still not much optimism on the soil moisture product.