Resume model training from checkpoint

[rusheb]

Description

Implement the capability to resume model training from a saved checkpoint. This should include adding an API for resuming a model from a checkpoint, which could be either downloaded from Weights & Biases (w&b) or stored locally. We may be able to leverage the w&b cache.

Not sure exactly when this will be required. Is it just when a training run crashes?

Acceptance Criteria

• Add an API for resuming model training from a checkpoint
• Support checkpointed models downloaded from w&b
• Support checkpointed models stored locally

[traeuker]

Happy to descope it as @afspies probably has the hackiest way to do this himself (joke), and others probably don't train models where this is worth implementing(?) Not sure, though.
@mivanit you want to eye this in the future?

[mivanit]

Probably 90% of the work required for this is done, so I think I might as well implement it.