Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fix: unable to find output_dir in multi-GPU during resume_from_checkpoint check #352

Merged
merged 3 commits into from
Sep 26, 2024
Merged

fix: unable to find output_dir in multi-GPU during resume_from_checkpoint check #352

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
3b9d780
fix: output_dir doesn't exist during resume_from_checkpoint
Abhishek-TAMU Sep 25, 2024
aed6de2
fix: fmt
Abhishek-TAMU Sep 25, 2024
928f500
Merge branch 'main' into output_dir
anhuong Sep 26, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions build/accelerate_launch.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -98,6 +98,8 @@ def main():
#
##########
output_dir = job_config.get("output_dir")
if not os.path.exists(output_dir):
Copy link
Collaborator

@anhuong anhuong Sep 26, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this solves the main case with running multi-GPU training from the image. But this issue would still appear if someone ran accelerate launch tuning/sft_trainer.py on their own but unsure if creating the directory within sft_trainer.py makes sense or if the same issue would come up where the dir may not be created....

After talking to Abhishek about this, adding this in before the separate process are initiated ensures the output_dir exists.

If we want only one process to check the resume from checkpoint condition then we can maybe add the check if dist.rank() == 0 to process resume_from_checkpoint similar to this case

But we want to check with others on this.

So for now, we are adding this improvement that fixes the bug and can look into the alternative as a future improvement.

os.makedirs(output_dir)
try:
# checkpoints outputted to tempdir, only final checkpoint copied to output dir
launch_command(args)
Expand Down
Loading