DSCI-621 Implementing the functionality to publish to HDX

[isatotun]

Jira ticket -> https://humanitarian.atlassian.net/browse/DSCI-621

[zackarno]

Some issues i've noticed on the staged files so far:

1. when I download the tabular stats the excel file does not open - it says it is corrupt
2. when I download the zipped raster file it goes into my download folder as a file that says: baseline_zipped_file.xlsx . The file is also grayed out and can't be opened. This should be a .zip file containing cloud optimized geotifs that can easily be unzipped.

with some finagling of tabular data set it can be opened as .csv. We've already discussed, but also noting here:

• that admin 2 stats tab is missing SFED_BASELINE and admin labels (only pcodes available).
• suggest changing colname from value to SFED
• admin 1 stats tab missing
• readme missing (i can provide a draft)

[isatotun]

Some issues i've noticed on the staged files so far:

1. when I download the tabular stats the excel file does not open - it says it is corrupt
2. when I download the zipped raster file it goes into my download folder as a file that says: baseline_zipped_file.xlsx . The file is also grayed out and can't be opened. This should be a .zip file containing cloud optimized geotifs that can easily be unzipped.

with some finagling of tabular data set it can be opened as .csv. We've already discussed, but also noting here:

• that admin 2 stats tab is missing SFED_BASELINE and admin labels (only pcodes available).
• suggest changing colname from value to SFED
• admin 1 stats tab missing
• readme missing (i can provide a draft)

These comments are related to an older version of the datasets.

[zackarno]

These comments are related to an older version of the datasets.

got it, thanks for pointing me to the updated data set in the jira ticket and explaining the caching/staging issues.

So some new comments:

Raster 90d zipped file: The zip file looks a bit strange. it appears to include the 90d prior to 2024-01-01 and just include 1 single band (SFED_BASELINE). This file is supposed to be the file that updates daily including the latest date. Each date/ cloud optimized geotif should include two bands 1. SFED, 2. SFED_BASELINE.

tabular excel file:

• looks better, opens without issues, and now see 3 tabs
• One small tweak would be to capitalize sfed_baseline to SFED_BASELINE to harmonize w/ SFED and RP . You can just modify the SQL in the function i provided.

[zackarno]

@isatotun - i checked the issue leonardo noted on the call for the missing SFED_BASELINE data in the tabular data set.

I can confirm that this bug is not introduced by the function i wrote to extract the baseline

pg.fs_rolling_11_day_mean() shown here. Therefore, I believe it is being introduced in this PR - my guess is a faulty join somewhere?

Just to note the function above does return some some NaN values but that is only when the original admin units have no data in the postgres db because they are too small for the zonal stats methodology used here. I think these NaN values will all be removed when we filter to only HRP countries and they don't account for the more widespread issue seen in the data on stage