Any support for peft methods on PPO?

[HaochenZhao]

I am working on a project about applying RL to LLM but only have very limited resource. Hope verl team can support peft methods like lora on your ppo trainer

[Jiayi-Pan]

Hi we are working on it, we'll try to send a PR in 2 weeks

[vermouth1992]

Hi we are working on it, we'll try to send a PR in 2 weeks

Shall we have a discussion on how this should be implemented?

[PeterSH6]

I think we can organize a quick Zoom meeting to discuss this. cc: @Jiayi-Pan

[corbt]

I'd also be interested in joining the meeting if it will happen in English. 🙂

[vermouth1992]

I'd also be interested in joining the meeting if it will happen in English. 🙂

Hi @corbt, we haven't started yet. But here is my draft. The implementation is pretty straight-forward

• We first apply peft to the hf model
• merge adaptor to the base model: https://huggingface.co/docs/peft/main/en/package_reference/tuners#peft.tuners.tuners_utils.BaseTuner.merge_adapter
• bind weights to vllm and generate
• unmerge adaptor https://huggingface.co/docs/peft/main/en/package_reference/tuners#peft.tuners.tuners_utils.BaseTuner.unmerge_adapter
• This can be implemented by introducing a peft sharding manager

[vermouth1992]

Another optimization that we can do is to use the base model for reference policy. This can also be done easily by using disable_adaptors API. https://huggingface.co/docs/transformers/main/en/peft

In this way, we only keep one model for actor/rollout/reference policy. The memory consumption of GRPO is identical to DPO

[Jiayi-Pan]

Hi I have some collaborators interested in using this as the first PR to veRL. It should be ready in 2 days