fix: (1) skipped last step (2) redundant validation and logging

[dirtyDan0]

This PR solves these 2 following problems.

1. Last step skipped

   self.global_steps += 1 before if self.global_steps >= self.total_training_steps makes the last step skipped.

   We start from step 1, and we expect self.total_training_steps in total.

   verl/verl/trainer/ppo/ray_trainer.py

   Lines 999 to 1001 in 82b38e2

   	self.global_steps += 1
   	if self.global_steps >= self.total_training_steps:

   When self.global_steps == self.total_training_steps-1:

   • we have only executed self.total_training_steps-1 steps

   • self.global_steps is updated to self.total_training_steps

   • self.global_steps >= self.total_training_steps is satisfied, and the training ends.

   Therefore, we should put self.global_steps += 1 at last

2. redundant validation and logging

   If self.total_training_steps % self.config.trainer.test_freq == 0 :

   • self._validate() will be executed twice

     1. verl/verl/trainer/ppo/ray_trainer.py

        Line 984 in 82b38e2

        val_metrics: dict = self._validate()

     2. verl/verl/trainer/ppo/ray_trainer.py

        Line 1005 in 82b38e2

        val_metrics = self._validate()

   • logging will also be executed twice

     1. verl/verl/trainer/ppo/ray_trainer.py

        Line 985 in 82b38e2

        metrics.update(val_metrics)

        and

        verl/verl/trainer/ppo/ray_trainer.py

        Line 997 in 82b38e2

        logger.log(data=metrics, step=self.global_steps)

     2. verl/verl/trainer/ppo/ray_trainer.py

        Line 1007 in 82b38e2

        logger.log(data=val_metrics, step=self.global_steps)

[CLAassistant]

All committers have signed the CLA.