Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Set evaluation epoch iterator object to None for each model.fit call #18659

Merged
merged 2 commits into from
Oct 23, 2023

Conversation

SuryanarayanaY
Copy link
Contributor

@SuryanarayanaY SuryanarayanaY commented Oct 20, 2023

When using model.fit() with validation_data, during first call to fit() it will create an EpochIterator object for evaluation and cache it. This cache will be deleted after completion of all epochs.

However if we try to change the validation_data shapes for any reason and called model.fit() again it will raise an exception like Graph execution error which is intended. After this if we call model.fit() with correct validation_data that of first training call it won't work.

The reason being is after first call is success evaluation EpochIterator will be deleted at the end and during second training call a new EpochIterator will be generated again with changed shape.But during evaluate() call it will raise an exception terminating the process without deleting evaluation EpochIterator object and it remains in cache. During third call though with correct validation_data it will not create new EpochIterator object as one already exists in cache which makes it fail again.

Hence proposing a fix for this with this commit.

Fixes #18653

@codecov-commenter
Copy link

codecov-commenter commented Oct 20, 2023

Codecov Report

Attention: Patch coverage is 0% with 2 lines in your changes missing coverage. Please review.

Project coverage is 57.73%. Comparing base (6e54f7f) to head (8a468cf).
Report is 1069 commits behind head on master.

Files with missing lines Patch % Lines
keras/backend/jax/trainer.py 0.00% 1 Missing ⚠️
keras/backend/tensorflow/trainer.py 0.00% 1 Missing ⚠️

❗ There is a different number of reports uploaded between BASE (6e54f7f) and HEAD (8a468cf). Click for more details.

HEAD has 6 uploads less than BASE
Flag BASE (6e54f7f) HEAD (8a468cf)
keras 4 1
keras-jax 1 0
keras-torch 1 0
keras-tensorflow 1 0
Additional details and impacted files
@@             Coverage Diff             @@
##           master   #18659       +/-   ##
===========================================
- Coverage   78.50%   57.73%   -20.77%     
===========================================
  Files         334      335        +1     
  Lines       32596    32946      +350     
  Branches     6376     6450       +74     
===========================================
- Hits        25589    19023     -6566     
- Misses       5452    12497     +7045     
+ Partials     1555     1426      -129     
Flag Coverage Δ
keras 57.73% <0.00%> (-20.66%) ⬇️
keras-jax ?
keras-numpy 57.73% <0.00%> (-0.22%) ⬇️
keras-tensorflow ?
keras-torch ?

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Copy link
Collaborator

@fchollet fchollet left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the fix -- it looks good to me. Please add a simple unit test for the failure case.

Copy link
Collaborator

@fchollet fchollet left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thank you

@google-ml-butler google-ml-butler bot added kokoro:force-run ready to pull Ready to be merged into the codebase labels Oct 23, 2023
@fchollet fchollet merged commit 5cce1b2 into keras-team:master Oct 23, 2023
6 checks passed
@google-ml-butler google-ml-butler bot removed ready to pull Ready to be merged into the codebase kokoro:force-run labels Oct 23, 2023
@SuryanarayanaY SuryanarayanaY deleted the fix-validation branch October 23, 2023 14:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Status: Merged
Development

Successfully merging this pull request may close these issues.

Unable to change validation dataset for "model.fit()" function after it raises an exception
5 participants