Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Model checkpoint overwriting should be smarter #10

Open
yawnston opened this issue Jun 13, 2022 · 0 comments
Open

Model checkpoint overwriting should be smarter #10

yawnston opened this issue Jun 13, 2022 · 0 comments
Labels
enhancement New feature or request

Comments

@yawnston
Copy link
Member

The model checkpoint file currently contains no information about whether it is the final version. As such, when loading the model from a checkpoint and hot reloading is enabled, the first iteration of hot reloading will overwrite the model checkpoint if overwriting is configured.

It would be nice if the overwriting was smarter:

  • If there existed no model checkpoint at startup, overwriting should be enabled for every iteration of hot reloading.
  • If there existed a model that was not final, overwriting should be enabled for every iteration of hot reloading.
  • If there existed a model that was final, either hot reloading should not run at all, or it should only overwrite the model checkpoint file on the last iteration.
@yawnston yawnston added the enhancement New feature or request label Jun 13, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant