DeepSpeed with multiple optimizer in pytorch ligthning

[MaugrimEP]

I'm trying to use deepspeed_stage_3_offload and deepspeed_stage_3 strategies to train a big stylegan model.

Before the start of the training, I get the error

pytorch_lightning.utilities.exceptions.MisconfigurationException: DeepSpeed currently only supports single optimizer, single optional scheduler.

But I need two optimizers for the GAN, is there a workaround?

[rohitgr7]

with a limitation with deepspeed that it only supports single optimizer
https://deepspeed.readthedocs.io/en/latest/initialize.html#deepspeed.initialize

[MaugrimEP]

Ho ok, I thought maybe a workaround would exist in lightning because they have a tutorial with a DCGAN and two optimizers. https://www.deepspeed.ai/tutorials/gan/

[SophieOstmeier]

Are there any updates on this?

[npuichigo]

@awaelchli