[BUG] pdsh runner doesn't work with tqdm bar

[Superskyyy]

Describe the bug
Training tqdm bar won't show in the rank0 console if pdsh is used as the launcher.

To Reproduce
Run any multi-node training on an example script, the tqdm bar from local rank (localhost) won't show up but show as a blank line.

I assume the root cause it pdsh tries to ssh into even localhost, then the bar cannot be logged through a ssh session somehow.

Expected behavior
Be able to see the tqdm bar.

System info (please complete the following information):

• OS: Ubuntu 22.04
• x8 GPUs each node
• Python version 3.12.8
• 4 Nodes, same network

Launcher context
PDSH

[Superskyyy]

huggingface/transformers#22689

[loadams]

Hi @Superskyyy - do you have a sample repro that I can run to use as a starting point for debug?

[Superskyyy]

Hi @Superskyyy - do you have a sample repro that I can run to use as a starting point for debug?

I guess you can directly use accelerate + deepspeed to run the ppo example in the official TRL repo. https://github.com/huggingface/trl/tree/main/examples/scripts/ppo but i suppose this problem will exist on all deepspeed + pdsh based training code.