Create issues to allow premet & spatials as input

[lisakaser]

resources:
https://nsidc.org/sites/default/files/documents/other/guidelines-preliminary-metadata-creation-and-data-product-delivery.pdf

(maybe helpful too...assuming the first is, indeed, helpful) https://nsidc.atlassian.net/wiki/spaces/TECHS/pages/51022884/Determining+Granule+Spatial+Representation+for+Data+Producer-Submitted+Metgenned+Data+Sets
IMPORTANT NOTE ABOUT THE ABOVE RESOURCES: These were written based on the needs of SIPSMetgen which required vertex order be specified in a clockwise (as viewed from above) order in .spo files. However, what MetGenC is doing (ordering vertices in a counter-clockwise order) in the ummg files output is correct (i.e. is what's required for CMR/Cumulus ingest success). In order for MetGenC to preserve current vertex order expectations for .spo files, it'll have to flip the order before incorporating them in the ummg file.

attachment fyi: I had to append ".txt" to the end of these file names to be allowed to paste them here!

IceBridge spatial file where all the lat/lon values are dumped from the science file (a .csv) and listed for metgen to treat as a point cloud around which it's to make a gpolygon; in this specific case the spatial representation happens to be more blob-like than a snake-like sock. The data set is specified as having a geodetic granule representation in CMR:
IDHDT4_2014-2011_ATM_dhdt_antarctica.csv.spatial.txt

Icebridge spatial where all lat/lon are dumped from the csv science file, but forms the OIB classic snake-like gpolygon.
IRMCR2_20191109_01.csv.spatial.txt

SMAP spatial file defining a bounding rectangle spatial rep for a netCDF file. The data set is specified as having a cartesian granule representation in CMR (bounding rectangles are the only spatial rep we use that's assigned to cartesian):
NSIDC-0774-EASE2_S3.125km-SMAP_Radar_Slice-2015186_2015193-1.2VV-M-SIR-v1.1.nc.spatial.txt

SnowEx spatial file defining a point spatial rep for a csv science file. The data set is specified as having a geodetic granule representation in CMR.
SNEX_Met_LSOS_final_output.csv.spatial.txt

Our good friend, 0081, which utilized the "spo" (for spatial polygon override) type of spatial file which signaled to metgen that the points in the file defined vertices for it to just transfer into the metadata file it generated. The data set is specified as having a geodetic granule representation in CMR.
NSIDC0081_SEAICE_PS_N25km_20240606_v2.0.spo.txt

Premet examples:

ITS_LIVE_velocity_120m_RGI01A_0000_v02_dvy_dt_20230308_20230315_v01.0_per-DSTEW-recos.premet.txt

IDHDT4_2011-2010_ATM_dhdt_antarctica_per-DSTEW-recos.csv.premet.txt

[afitzgerrell]

@juliacollins I added this into the issue description, but I think it's comment worthy:

Regarding the resources linked in the description: They were written based on SIPSMetgen which required vertex order be specified in a clockwise (as viewed from above) order in .spo files. However, what MetGenC is doing (ordering vertices in a counter-clockwise order) in the ummg files output is correct (i.e. is what's required for CMR/Cumulus ingest success)!! When an operator feeds MetGenC an input file where the coordinate pairs listed are meant to define a gpolygon's vertices, they should be listed in counter-clockwise order.

[lisakaser]

Issues created:

• Update ini options to specify location for premet and spatial ancillary files #155 Update ini options to specify location for premet and spatial ancillary files (2SP)
• Use temporal information from premet file in metgenc processing pipeline #160 Read premet and output info in UMM-G (refactoring story needs to be done first Review metgenc NetCDF & CSV readers and unify their configuration for the operators #109) (2SP)
• Allow ThemeID and 'Campaigns' -> UMM-G (large story) Include .premet additional attributes (e.g. campaigns) in UMM-G output #162
• Read spo (polygon) or spatial (can be large file, can be points to generate socks around or 4 corner points) (2SP) Read spatial information from a .spatial file associated with a granule #161 (limited the story to .spatial; I think it makes sense to deal with .spo entirely in Correct point order in spo file before creating polygon #158 )
• Correct point order in spo file before creating polygon #158 Correct point order in spo file before creating polygon - these are lat/lon values - and close the polygon (2SP)
  -> after all these issues MetGenC should already know what to do with the spatial info retrieved from .spatial or .spo

[juliacollins]

@juliacollins I added this into the issue description, but I think it's comment worthy:

Regarding the resources linked in the description: They were written based on SIPSMetgen which required vertex order be specified in a clockwise (as viewed from above) order in .spo files. However, what MetGenC is doing (ordering vertices in a counter-clockwise order) in the ummg files output is correct (i.e. is what's required for CMR/Cumulus ingest success)!! When an operator feeds MetGenC an input file where the coordinate pairs listed are meant to define a gpolygon's vertices, they should be listed in counter-clockwise order.

As noted in our meeting, I think the right strategy is for metgenc to read the .spo file contents as they are currently defined, then update the values to create a counter-clockwise-specified polygon.

[afitzgerrell]

@juliacollins, I've found that the premet values: LocalVersionID and LocalGranuleID appear to be moot for umm-g / Cumulus ingest. This is based on finding that the .xml metadata files (able to be downloaded from EDSC with data archived in ECS collections*) contain the following:

         <ECSDataGranule>
            <SizeMBECSDataGranule>1.84104</SizeMBECSDataGranule>
            <LocalGranuleID>IRWIS2_Data_20120325.csv</LocalGranuleID>
            <ProductionDateTime>2013-11-07 09:18:45.000</ProductionDateTime>
            <LocalVersionID>001</LocalVersionID>
        </ECSDataGranule>

In this morning's planning meeting I think you and Kevin already keyed into LocalGranuleID likely being irrelevant since a regex will generate the Identifiers/Identifier value for a multi-file granule for MetGenC which'll fulfilling the LocalGranuleID's former role in SIPSMetgen for ECS ingest.

I don't see a comparable field for LocalVersionID. The above ECSDataGranule block isn't included in the file-level metadata options available under "more actions ->view info" for a granule in EDSC; it was never used by external data producers; Operators weren't wont to change it; and further, *it looks like data no option to download file-level metadata with data exists in EDSC cloud collections.

If data producers (LVIS) continue to include LocalVersionID & LocalGranuleID in premets, MetGenC should skip 'em (although if it wouldn't hurt, maybe we could let LVIS know that they're not necessary and see what they say?).

[juliacollins]

@lisakaser I think we have the necessary issues in place to describe premet/spatial/spo support:
#155
#160
#161
#158
#162