DDP specifics: Single node function execution, test loss sync, num_workers, sync_batchnorm, precision

[ManiadisG]

❓ Questions and Help

What is your question?

This is about DDP specifics and about the handling of functions in a script that we only want executed once (not for every GPU).

I think they are missing from the doc and I couldn't find answers elsewhere. I apologize if they have been covered already.

Questions:

1. Does the regular torch.save(model.state_dict(), path) call work normally or does DDP complicate things for multiple GPUS?
2. Does DDP run all the functions of a script for every GPU? For example, in the following script will the delete_and_remake_folder function be executed multiple times (that would result in conflicts)? Is there a way to specify functions to be run only once?
3. Am I correct that sync_batchnorm=True and precision=16 work in DDP?
4. Does the trainer.test() function automatically aggregate results accross devices or is it required to set self.log(loss, sync_dist=True) in the model?
5. Am I correct in assuming that, if we set num_workers=X in a Dataloader, the actual CPU core usage will be X*N for N GPUS?

Questions 1-4 are summarized in the following script and whether it works/can work.

def main():
   delete_and_remake_folder() # I only want to run once
   model = Model()
   trainer = Trainer(gpus = 8, backend='ddp, sync_batchnorm=True, precision=16)
   trainer.fit()
   trainer.test()
   torch.save(model.pt_model.state_dict(), save_dir) # I probably only want to run once (?)

[s-rog]

1. yes, but you might run into the issue in your 2nd question
2. yes, yes if self.global_rank == 0: do stuff
3. yes
4. no, yes (think of it like another val loop)
5. yes

I'll close this for now, if you have more questions please use our forums! https://forums.pytorchlightning.ai/