fix: unable to find output_dir in multi-GPU during resume_from_checkpoint check

[Abhishek-TAMU]

Description of the change

Addition of code to create output_dir in accelerate_launch.py if it doesn't exist.

Related issue number

#1352

How to verify the PR

Run full fine tuning or LoRA tuning with multiple GPUs and check if issue exists.

Was the PR tested

• I have added >=1 unit test(s) for every new method I have added.
• I have ensured all unit tests pass

[github-actions]

Thanks for making a pull request! 😃
One of the maintainers will review and advise on the next steps.

[github-actions]

Thanks for making a pull request! 😃
One of the maintainers will review and advise on the next steps.

[anhuong]

I think this solves the main case with running multi-GPU training from the image. But this issue would still appear if someone ran accelerate launch tuning/sft_trainer.py on their own but unsure if creating the directory within sft_trainer.py makes sense or if the same issue would come up where the dir may not be created....

After talking to Abhishek about this, adding this in before the separate process are initiated ensures the output_dir exists.

If we want only one process to check the resume from checkpoint condition then we can maybe add the check if dist.rank() == 0 to process resume_from_checkpoint similar to this case

But we want to check with others on this.

So for now, we are adding this improvement that fixes the bug and can look into the alternative as a future improvement.