Partial accelerator.prepare() Usage

[zhanglixuan0720]

I am working with a model that consists of two separate encoders (one for RGB and one for Depth) and an MLP head for action prediction. The outputs of the two encoders are concatenated before being passed into the MLP head. I am using accelerator.prepare() to prepare only the two encoders and the MLP head, but the concatenation (torch.cat) is not included in prepare(). I am wondering how this affects distributed training. Specifically:

1. Will the concatenation operation run on a single GPU if it is not prepared?
2. Could this cause device mismatches between prepared and unprepared module?
3. How does gradient synchronization behave in this scenario?

Thanks in advance for your help!