Point reproduce in Python segments to HuggingFace URLs #2316
Conversation
|
The latest updates on your projects. Learn more about Vercel for Git ↗︎
|
There was a problem hiding this comment.
Hey @nikhilwoodruff, thanks for this quick turnaround. My main thought is in my comment below, but TL;DR, would it be more logical (and equally as fast) to solve this in us-data? Looking forward to your thoughts.
| @@ -189,9 +189,9 @@ export function getImplementationCode( | |||
| if (hasDatasetSpecified) { | |||
| datasetText = DEFAULT_DATASETS[dataset]; | |||
| } else if (isState) { | |||
| datasetText = "pooled_3_year_cps_2023"; | |||
| datasetText = "hf://policyengine/policyengine-us-data/pooled_3_year_cps_2023.h5"; | |||
There was a problem hiding this comment.
question, blocking: Would it be more sustainable to modify this setting in us-data and keep the app the same?
It seems to me we always want the API and country packages both to prefer HuggingFace datasets over GitHub. It also seems like it'd be relatively simple to modify us-data to either recognize this preference, or just only download from once source or the other, instead of our current method where we still send data updates to both GitHub and HuggingFace.
Could we just modify us-data to do this? Seems unwieldy to ask users to remember this entire link.
There was a problem hiding this comment.
As per the policyengine.py design doc, I think we should move away from us-data being a package dependency so I'd vote against this.
In the medium term we'll move to policyengine.py, which abstracts the huggingface URL away behind constants, and update the reproduce in python segment accordingly. Right now the reproduce in python segment is broken without this, so I'd suggest still keeping this solution in the -app repo here.
Fixes #2315
Description
Point dataset strings to URLs instead
Changes
As above.
Screenshots
No tests added.