Accelerate launch will hang when the training loop data loader have a file not found error (even though this is catched)

[lucasjinreal]

In training in LLaVA 1.6, typically we might unevitable image file missing, it has this catching logic:

def __getitem__(self, i) -> Dict[str, torch.Tensor]:
        attempt, max_attempt = 0, 10
        while attempt < max_attempt:
            try:
                # sample an item
                data_dict = self._sample_item(i)
                # if data_dict is not None:
                break
            except Exception as e:
                attempt += 1
                print(f"Error in loading {i}, retrying...")
                import traceback

                print(e)
                traceback.print_exc()
                i = random.randint(0, len(self.list_data_dict) - 1)
        return data_dict

Actually works fine when using deepspeed train.py or torchrun train.py.

But when using accelerate launch, especially with auto_find_batch_size = True, it will hang when file not found but catched.

Am not sure how to make it work with accelerate since i found accelerate saves memory than deepspeed so I had to use it

[KeshavSingh29]

This is more of a programming issue on your end than issue with accelerate.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.