Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ENH: Refactor filters into filtering #71

Draft
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

jhlegarreta
Copy link
Contributor

@jhlegarreta jhlegarreta commented Jan 25, 2025

Refactor code blocks that perform filtering operations within models into the filtering module so that the model and filter concepts are separated, supporting more cleanly pipelining models with filters within the new Estimator philosophy

Add the percentile-based detrending feature that was temporarily removed in commit 7322e93.

Closes #63.

@jhlegarreta
Copy link
Contributor Author

A few comments:

@jhlegarreta jhlegarreta force-pushed the RefactorFilteringCode branch from 5cec644 to 7e67f76 Compare January 25, 2025 19:17
reference = np.percentile(centers[centers >= 1.0], DEFAULT_CLIP_PERCENTILE)
centers[centers < 1.0] = reference
drift = reference / centers
return shelldata * drift
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Naming for above two functions/filters should be improved.

Refactor code blocks that perform filtering operations within models
into the `filtering` module so that the model and filter concepts are
separated, supporting more cleanly pipelining models with filters within
the new `Estimator` philosophy.

Add the percentile-based detrending feature that was temporarily removed
in commit 7322e93.
@jhlegarreta jhlegarreta force-pushed the RefactorFilteringCode branch from 1170199 to 504ea30 Compare January 25, 2025 19:30

clipped_dataset._dataset = clipped_dataset._dataset.dataobj[..., shellmask]

return clipped_dataset
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we clip the data, maybe the gradients stored in the datasets should also be clipped.

Copy link

codecov bot commented Jan 25, 2025

Codecov Report

Attention: Patch coverage is 29.03226% with 22 lines in your changes missing coverage. Please review.

Project coverage is 67.35%. Comparing base (7237ecf) to head (504ea30).

Files with missing lines Patch % Lines
src/nifreeze/data/filtering.py 18.51% 22 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main      #71      +/-   ##
==========================================
- Coverage   68.75%   67.35%   -1.40%     
==========================================
  Files          20       20              
  Lines         957      965       +8     
  Branches      121      120       -1     
==========================================
- Hits          658      650       -8     
- Misses        254      271      +17     
+ Partials       45       44       -1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Move detrending from AverageModel to its own Filter
1 participant