feature: option to load wind and solar resource data from HPC #414

elenya-grant · 2025-01-09T00:18:41Z

Load wind and solar resource data from h5 datasets

Added the option and functionality to load wind and solar resource data from Wind Toolkit and NSRDB data files (usually hosted on HPC but can be downloaded locally too). This new feature is primarily based on work previously done by @dakotaramos. To use this functionality, the user must input renewable_resource_origin = HPC when creating the SiteInfo object. Did not add new tests because this can only be used if the h5 type datafile are downloaded locally or code is run on an HPC that hosts these datasets.

Also fixed bug in SiteInfo when setting year from input argument rather than from the input data dictionary.

PR Checklist

RELEASE.md has been updated to describe the changes made in this PR
Documentation
- Docstrings are up-to-date
- Related docs/ files are up-to-date, or added when necessary
- Documentation has been rebuilt successfully
- [-] Examples have been updated
Tests pass (If not, and this is expected, please elaborate in the tests section)
PR description thoroughly describes the new feature, bug fix, etc.

Related issues

Impacted areas of the software

hopp/simulation/technologies/resource/wind_toolkit_data.py
- HPCWindData: new Resource type class to load wind resource data from Wind Toolkit data files hosted on HPC
hopp/simulation/technologies/resource/nsrdb_data.py
- HPCSolarData: new Resource type class to load solar resource data from NSRDB data files hosted on HPC
hopp/simulation/technologies/resource/__init__.py
- added imports for HPCSolarData and HPCWindData
hopp/simulation/sites/site_info.py
- added renewable_resource_origin attribute to specify whether to download resource data from API call or load from HPC datasets
- changed solar_resource and wind_resource to optional inputs to prevent redundant resource file loading
- added wtk_source_path and nsrdb_source_path as optional inputs to specify WTK or NSRDB data set directories
- fixed bug that sets resource year to 2012 if 'year' isn't in 'data' dictionary but input to SiteInfo as argument
pyproject.toml
- added NREL-rex to dependencies list

Additional supporting information

Test results, if applicable

…dated upstream functions

…year being set to 2012

… and updated doc strings

dakotaramos · 2025-01-14T18:40:59Z

hopp/simulation/technologies/resource/nsrdb_data.py

+            nsrdb_source_path (Union[str,Path], optional): directory where NSRDB data is hosted on HPC. Defaults to "".
+            filepath (str, optional): filepath to NSRDB h5 file on HPC. Defaults to "".
+                - should be formatted as: /path/to/file/name_of_file.h5
+        """


@elenya-grant in the windtoolkit file you have verbiage for Raises: FileNotFoundError: in the docstrings. For consistency I'd add it into this files docstrings as well.

fixed this!

dakotaramos · 2025-01-14T18:46:51Z

hopp/simulation/technologies/resource/wind_toolkit_data.py

+
+        if filepath == "" and wtk_source_path=="":
+            # use default filepaths based on resource year
+            if self.year < 2014:


This all looks good, I had a thought when looking at the if, elif statements for the year < or = 2014. Is it worth raising an error if the year is > 2014? I don't believe the normal WindData class has that functionality, but could be a good item to add just in case? no strong feelings either way.

I added a value error if the resource year is outside of valid options for that dataset (also added tests for this)

dakotaramos · 2025-01-14T18:51:21Z

This looks good to me, made two comments with possible additions / edits, other than that I think this can be approved and merged.

…rce data

dakotaramos

Good to merge from my perspective, all comments from previous review have been addressed.

johnjasa · 2025-01-17T20:02:13Z

tests/hopp/test_resource_download.py

+def test_wtk_resource_filenotfound():
+    wtk_fake_dir = str(ROOT_DIR)
+    resource_year = 2012
+    wtk_fake_fpath = os.path.join(str(ROOT_DIR),f"wtk_conus_{resource_year}.h5")
+    with pytest.raises(FileNotFoundError) as err:
+        HPCWindData(lat = 35.201, lon = -101.945, year = resource_year, wind_turbine_hub_ht = 110, wtk_source_path=wtk_fake_dir)
+    assert str(err.value) == f"Cannot find Wind Toolkit .h5 file, filepath {wtk_fake_fpath} does not exist"
+
+def test_wtk_resource_invalid_year():
+    wtk_fake_dir = str(ROOT_DIR)
+    resource_year = 2006
+    wtk_fake_fpath = os.path.join(str(ROOT_DIR),f"wtk_conus_{resource_year}.h5")
+    with pytest.raises(ValueError) as err:
+        HPCWindData(lat = 35.201, lon = -101.945, year = resource_year, wind_turbine_hub_ht = 110, wtk_source_path=wtk_fake_dir)
+    assert str(err.value) == f"Resource year for WIND Toolkit Data must be between 2007 and 2014 but {resource_year} was provided"
+
+def test_nsrdb_resource_filenotfound():
+    nsrdb_fake_dir = str(ROOT_DIR)
+    resource_year = 2012
+    nsrdb_fake_fpath = os.path.join(str(ROOT_DIR),f"nsrdb_{resource_year}.h5")
+    with pytest.raises(FileNotFoundError) as err:
+        HPCSolarData(lat = 35.201, lon = -101.945, year = resource_year, nsrdb_source_path=nsrdb_fake_dir)
+    assert str(err.value) == f"Cannot find NSRDB .h5 file, filepath {nsrdb_fake_fpath} does not exist"
+
+def test_nsrdb_resource_invalid_year():
+    nsrdb_fake_dir = str(ROOT_DIR)
+    resource_year = 2023
+    nsrdb_fake_fpath = os.path.join(str(ROOT_DIR),f"nsrdb_{resource_year}.h5")
+    with pytest.raises(ValueError) as err:
+        HPCSolarData(lat = 35.201, lon = -101.945, year = resource_year, nsrdb_source_path=nsrdb_fake_dir)
+    assert str(err.value) == f"Resource year for NSRDB Data must be between 1998 and 2022 but {resource_year} was provided"


Love these tests, thank you!

RHammond2

Most of my comments are recommendations, including those involving the changes to the initialization routines to allow for more testing. As it stands this looks pretty good to me and could be merged as-is.

RHammond2 · 2025-01-17T23:00:33Z

hopp/simulation/technologies/resource/nsrdb_data.py

+NSRDB_DEP = "/datasets/NSRDB/deprecated_v3/nsrdb_"
+
+# NOTE: Current version of PSM v3.2.2 which corresponds to /api/nsrdb/v2/solar/psm3-2-2-download 
+NSRDB_NEW = "/datasets/NSRDB/current/nsrdb_"


I'd recommend converting these to Path objects (Path("file_path")) to make some later manipulations easier.

RHammond2 · 2025-01-17T23:03:26Z

hopp/simulation/technologies/resource/nsrdb_data.py

+
+        if filepath == "" and nsrdb_source_path=="":
+            # use default filepath
+            self.nsrdb_file = NSRDB_NEW + "{}.h5".format(self.year)


I'd recommend the use of f-strings in cases like this so it reads a bit easier. f"{self.year}.h5". If you decide to make NSRDB_NEW a path, this becomes NSRDB_NEW / f"{self.year}.h5" as an example

RHammond2 · 2025-01-17T23:05:55Z

hopp/simulation/technologies/resource/nsrdb_data.py

+            self.nsrdb_file = filepath
+        elif filepath == "" and nsrdb_source_path != "":
+            # directory of h5 files (nsrdb_source_path) is provided by user
+            self.nsrdb_file = os.path.join(str(nsrdb_source_path),"nsrdb_{}.h5".format(self.year))


Similarly, for using path this would look like: Path(nsrdb_source_path) / f"nsrdb_{self.year}.h5". I also passed nsrdb_source_path to Path() because it could either be a Path or string, but passing a Path object to Path again won't cause any errors, so it's not worth checking first.

RHammond2 · 2025-01-17T23:07:33Z

hopp/simulation/technologies/resource/nsrdb_data.py

+            # extract remaining datapoints: year, month, day, hour, minute, dn, df, gh, wspd,tdry, pres, tdew
+            # NOTE: datasets have readings at 0 and 30 minutes each hour, HOPP/SAM workflow requires only 30 minute reading values -> filter 0 minute readings with [1::2]
+            # NOTE: datasets are not auto shifted by timezone offset -> wrap extraction in SAMResource.roll_timeseries(input_array, timezone, #steps in an hour=1) to roll timezones
+            # NOTE: solar_resource.py code references solar_zenith_angle and RH = relative_humidity but I couldn't find them actually being utilized. Captured them below just in case.


Could you move these extra long comments to multiple lines? I know we don't have any enforcement on this, but it's nice to be able to read comments without sideways scrolling.

RHammond2 · 2025-01-17T23:08:52Z

hopp/simulation/technologies/resource/nsrdb_data.py

+
+        # round to desired precision and convert to desired data type
+        # NOTE: unsure if SAM/PySAM is sensitive to data types and decimal precision. 
+        # If not sensitive, can remove .astype() and round() to increase computational efficiency


Is there a good way to find out? It would be nice to have it settled prior to merging, but if this is pretty small potatoes, I also get it.

RHammond2 · 2025-01-17T23:10:01Z

hopp/simulation/technologies/resource/wind_toolkit_data.py

+import os
+from hopp.simulation.technologies.resource.resource import Resource
+
+WTK_V10_BASE = "/datasets/WIND/conus/v1.0.0/wtk_conus_"


Same comment about Path: you don't need to but it's recommended to use them in place of strings to make manipulations easier.

I have not done this yet because the base paths include the first part of the filename as well. I am also not sure whether rex (the tool used to load the resource data from .h5 files) is compatible with Path objects or if it requires strings (I will need to check on this).

RHammond2 · 2025-01-17T23:19:39Z

hopp/simulation/technologies/resource/wind_toolkit_data.py

+        self.format_data() 
+
+
+    def calculate_heights_to_download(self):


This is an example of a good method to check in the test suite. Basically anything leading up to the download_resources() because we won't have access in the CI pipeline.

this test should be added when the ability to not download resource data is added to these classes (to avoid the file not found error)

RHammond2 · 2025-01-17T23:23:36Z

hopp/simulation/technologies/resource/wind_toolkit_data.py

+        self.download_resource()
+
+        # Set wind resource data into SAM/PySAM digestible format
+        self.format_data() 


Do we want these in the __init__() method? I would assume it's that it's initialized, then downloaded, or would this pattern be more helpful to the intended workflow? An alternative is to have a flag in the __init__ that triggers downloading in the initialization or not, such as download: bool = True. This would also for passing false and testing paths and other surrounding items for correctness.

I agree with you. I have these in the __init__() method because it follows the same workflow as solar_resource.py and wind_resource.py. I think that if we want this change - this may be better suited for PR #415 since that includes the ability to initialize solar_resource.py and wind_resource.py with resource data (and therefore it's not downloaded). Or - I think this could be part of a future PR that includes other development to SiteInfo and the resource functions.

I created an issue (issue #423) to address this

RHammond2 · 2025-01-17T23:23:37Z

hopp/simulation/technologies/resource/wind_toolkit_data.py

+        if filepath == "" and wtk_source_path=="":
+            # use default filepaths based on resource year
+            if self.year < 2014 and self.year>=2007:
+                self.wtk_file = WTK_V10_BASE + "{}.h5".format(self.year)
+            elif self.year == 2014:
+                self.wtk_file = WTK_V11_BASE + "{}.h5".format(self.year)
+        elif filepath != "" and wtk_source_path == "":
+            # filepath (full h5 filepath) is provided by user
+            if ".h5" not in filepath:
+                filepath = filepath + ".h5"
+            self.wtk_file = filepath
+        elif filepath == "" and wtk_source_path != "":
+            # directory of h5 files (wtk_source_path) is provided by user
+            self.wtk_file = os.path.join(str(wtk_source_path),"wtk_conus_{}.h5".format(self.year))
+        else:
+            # use default filepaths
+            if self.year < 2014:
+                self.wtk_file = WTK_V10_BASE + "{}.h5".format(self.year)
+            elif self.year == 2014:
+                self.wtk_file = WTK_V11_BASE + "{}.h5".format(self.year)
+
+        # Check for valid filepath for Wind Toolkit file
+        if not os.path.isfile(self.wtk_file):
+            raise FileNotFoundError(f"Cannot find Wind Toolkit .h5 file, filepath {self.wtk_file} does not exist")


Checking the produced file paths are what's expected would also be helpful.

I think this is partially accomplished in the tests starting with test_wtk_resource_filenotfound_ and test_nsrdb_resource_filenotfound_. This would also be easier to do if (like you suggested) - resource data was not automatically loaded in the __init__() function which will always throw a FileNotFound error when running tests locally

RHammond2 · 2025-01-17T23:24:17Z

hopp/simulation/technologies/resource/nsrdb_data.py

+        if not os.path.isfile(self.nsrdb_file):
+            raise FileNotFoundError(f"Cannot find NSRDB .h5 file, filepath {self.nsrdb_file} does not exist")
+
+        # Pull data from HPC NSRDB dataset
+        self.download_resource()
+
+        # Set solar resource data into SAM/PySAM digestible format
+        self.format_data()


Same comment on these that I left in the WTK class.

added ability to grab wind and solar resource data off the HPC and up…

ee52ab9

…dated upstream functions

elenya-grant changed the base branch from main to develop January 9, 2025 00:19

elenya-grant added 6 commits January 9, 2025 10:36

updated doc strings and comments for wtk_data and nsrdb_data scripts

ea4a235

added comments to site_info for added options, fixed bug of resource …

03a4203

…year being set to 2012

updated formatting and docstrings for previous commit

b225d9a

updated RELEASE.md with recent changes

0a6fce3

changed input variable name in HPCWindData to align with WindResource…

721c43e

… and updated doc strings

added high level test for hpc resource data

a84edab

elenya-grant assigned elenya-grant and dakotaramos and unassigned elenya-grant Jan 10, 2025

elenya-grant marked this pull request as ready for review January 10, 2025 02:07

elenya-grant unassigned dakotaramos Jan 10, 2025

elenya-grant requested review from dakotaramos and kbrunik January 10, 2025 17:02

elenya-grant mentioned this pull request Jan 10, 2025

feature: option to initialize site_info with resource data #415

Open

7 tasks

elenya-grant requested review from RHammond2 and removed request for kbrunik January 13, 2025 20:49

dakotaramos reviewed Jan 14, 2025

View reviewed changes

elenya-grant added 3 commits January 15, 2025 12:30

added sphinx config path in readthedocs to fix build

83ac5a0

updated doc strings and minor updates to hpc resource functions

e26cb77

updated docstrings and added valid year check and tests for hpc resou…

a1f4f68

…rce data

elenya-grant requested a review from dakotaramos January 15, 2025 19:37

dakotaramos approved these changes Jan 15, 2025

View reviewed changes

johnjasa marked this pull request as draft January 16, 2025 20:44

elenya-grant added 3 commits January 16, 2025 16:04

changed site_info api doc file to markdown

6908c5c

added doc files for wind and solar resource

7e7d443

minor updates to doc strings in hpc resource classes

1de8e5e

merged NREL/develop into v3/hopp_dev and resolved conflicts

1047202

elenya-grant marked this pull request as ready for review January 17, 2025 00:35

johnjasa reviewed Jan 17, 2025

View reviewed changes

RHammond2 approved these changes Jan 17, 2025

View reviewed changes

elenya-grant added 2 commits January 20, 2025 15:50

updated string formatting

3019045

added tests for wtk and nsrdb resource if filepath is provided

4740b1d

elenya-grant mentioned this pull request Jan 20, 2025

Feature request: add ability to not automatically download or load resource data to enable more thorough testing #423

Open

split long comments into multiple lines

f7c7f79

elenya-grant merged commit 6839320 into NREL:develop Jan 22, 2025
4 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feature: option to load wind and solar resource data from HPC #414

feature: option to load wind and solar resource data from HPC #414

elenya-grant commented Jan 9, 2025 •

edited

Loading

dakotaramos Jan 14, 2025

elenya-grant Jan 15, 2025

dakotaramos Jan 14, 2025

elenya-grant Jan 15, 2025

dakotaramos commented Jan 14, 2025

dakotaramos left a comment

johnjasa Jan 17, 2025

RHammond2 left a comment

RHammond2 Jan 17, 2025

RHammond2 Jan 17, 2025

RHammond2 Jan 17, 2025

RHammond2 Jan 17, 2025

RHammond2 Jan 17, 2025

RHammond2 Jan 17, 2025

elenya-grant Jan 20, 2025

RHammond2 Jan 17, 2025

elenya-grant Jan 20, 2025

RHammond2 Jan 17, 2025

elenya-grant Jan 20, 2025 •

edited

Loading

RHammond2 Jan 17, 2025

elenya-grant Jan 20, 2025 •

edited

Loading

RHammond2 Jan 17, 2025

feature: option to load wind and solar resource data from HPC #414

feature: option to load wind and solar resource data from HPC #414

Conversation

elenya-grant commented Jan 9, 2025 • edited Loading

Load wind and solar resource data from h5 datasets

PR Checklist

Related issues

Impacted areas of the software

Additional supporting information

Test results, if applicable

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dakotaramos commented Jan 14, 2025

dakotaramos left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

RHammond2 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

elenya-grant Jan 20, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

elenya-grant Jan 20, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

elenya-grant commented Jan 9, 2025 •

edited

Loading

elenya-grant Jan 20, 2025 •

edited

Loading

elenya-grant Jan 20, 2025 •

edited

Loading