Floodscan stats #11
Merged
Floodscan stats #11
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
# Conflicts: # run_raster_stats.py # src/config/seas5.yml # src/config/settings.py # src/utils/cog_utils.py
Floodscan stats hannah
isatotun
reviewed
Nov 12, 2024
isatotun
reviewed
Nov 12, 2024
isatotun
reviewed
Nov 12, 2024
hannahker
commented
Nov 14, 2024
| """Test basic upsampling of a 2D dataset.""" | ||
| target_res = 0.5 # Upsample from 1.0 to 0.5 degrees | ||
| result = upsample_raster(simple_dataset, resampled_resolution=target_res) | ||
| result = upsample_raster(dataset_with_time, resampled_resolution=target_res) |
hannahker
commented
Nov 14, 2024
| start_date: 2023-12-01 | ||
| end_date: 2024-01-31 | ||
| iso3s: ["ETH"] | ||
| coverage: ["DZA", "AGO", "BEN", "BWA", "BFA", "BDI", "CPV", "CMR", "CAF", "TCD", "COM", "COG", "CIV", "CAP", "DJI", "EGY", "GNQ", "ERI", "SWZ", "ETH", "GAB", "GMB", "GHA", "GIN", "GNB", "KEN", "LS0", "LBR", "LBY", "MDG", "MWI", "MLI", "MRT", "MUS", "MAR", "MOZ", "NAM", "NER", "NGA", "RWA", "STP", "SEN", "SYC", "SLE", "SOM", "ZAF", "SSD", "SDN", "TGO", "TUN", "UGA", "TZA", "ZMB", "ZWE"] |
There was a problem hiding this comment.
Also a nice way to do this!
hannahker
commented
Nov 14, 2024
| dim = next(iter(extra_dim)) | ||
| if extra_dim[dim] == "str": | ||
| parsed_extra_dims[dim] = VARCHAR | ||
| else: |
There was a problem hiding this comment.
In the future, we'll probably want to have other data types, but can leave as is for now
hannahker
commented
Nov 14, 2024
| @@ -9,7 +9,7 @@ def cli_args(): | |||
| parser.add_argument( | |||
| "dataset", | |||
| help="Dataset for which to calculate raster stats", | |||
| choices=["seas5", "era5", "imerg"], | |||
| choices=["seas5", "era5", "imerg", "floodscan"], | |||
There was a problem hiding this comment.
Also as discussed, in the future it'd be good to think about how we can avoid the hard-coding of each dataset (since this happens in numerous places throughout the code). Although really more of a note for myself since I set it up this way...
hannahker
commented
Nov 14, 2024
|
|
||
| for dataset in pipelines: | ||
| config = load_pipeline_config(dataset) | ||
| if "coverage" in config: |
There was a problem hiding this comment.
Would be worth adding at least a comment in here to remind ourselves that we're assuming that the coverage is specified in ISO3 values.
|
@isatotun looks good to merge to me! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
WIP to generate raster stats for new Floodscan COG outputs. Since the Floodscan COGs include bands for both SFED and MFED data, I've generalized a bunch of functionality initially developed to handle the SEAS5
leadtimedimension. With these changes, the raster stats pipeline should be able to handle any generic 4th dimension (eg.bandorleadtime) coming from the input data. To properly create the Postgres table, this dimension just needs to be specified in the dataset'sconfig.ymlfile (see theextra_dimsproperty).To do:
floodscan) to theiso3table, which will tell us which ISO3s are covered. So any future datasets that have non-global coverage could get their own similar column.banddimension. This is currently just being pulled from the source COG, where values are either 1 or 2. We need to map these toSFEDandMFEDso that the output table is more informative.