-
Notifications
You must be signed in to change notification settings - Fork 19.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Set evaluation epoch iterator object to None for each model.fit call #18659
Conversation
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #18659 +/- ##
===========================================
- Coverage 78.50% 57.73% -20.77%
===========================================
Files 334 335 +1
Lines 32596 32946 +350
Branches 6376 6450 +74
===========================================
- Hits 25589 19023 -6566
- Misses 5452 12497 +7045
+ Partials 1555 1426 -129
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for the fix -- it looks good to me. Please add a simple unit test for the failure case.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM, thank you
When using
model.fit()
withvalidation_data
, during first call tofit()
it will create anEpochIterator
object for evaluation and cache it. This cache will be deleted after completion of all epochs.However if we try to change the validation_data shapes for any reason and called
model.fit()
again it will raise an exception like Graph execution error which is intended. After this if we callmodel.fit()
with correctvalidation_data
that of first training call it won't work.The reason being is after first call is success evaluation
EpochIterator
will be deleted at the end and during second training call a newEpochIterator
will be generated again with changed shape.But duringevaluate()
call it will raise an exception terminating the process without deleting evaluationEpochIterator
object and it remains in cache. During third call though with correct validation_data it will not create newEpochIterator
object as one already exists in cache which makes it fail again.Hence proposing a fix for this with this commit.
Fixes #18653