[Question] Ray tune with rsl_rl

[giulioturrisi]

Hello,
I'm not able to run ray tuner with rsl_rl. For example, the hydra parameters are not 1 to 1 with the one modified by the tuner as default (rl_games), and modifying the script, I'm not able to get the result of the run from the rsl library. Has anyone had success with it, or running wandb sweep?

Thanks

[glvov-bdai]

EDIT:

Hi, thanks for bringing this to my attention #1717 should work now for rsl_rl (see the changes I made related to logging). It will hopefully be merged up soon, but for now you should be able to use that fork/branch. You will need to write your own JobCfg, but logging should now work.

[giulioturrisi]

EDIT: With RSL (and using your branch), the trials end very quickly. Unclear which is the cause of this behavior (with the old version, I was able to run each trial until the end, and after I was getting the error). If I run the "invocation commands" that ray_tune prompt, everything run normally. So there should not be any error in the hydra params

I'm using the standard anymal env, these are the files that I use for the tuning.
https://drive.google.com/drive/folders/1NzZ7EpNdCC9XF4Ww6GtXJhLvR3z0Ho5S?usp=sharing

and to run I use the following commands:

echo "import ray; ray.init(); import time; [time.sleep(10) for _ in iter(int, 1)]" | ./isaaclab.sh -p (TERMINAL 1)

./isaaclab.sh -p source/standalone/workflows/ray/tuner.py --run_mode local --cfg_file source/standalone/workflows/ray/hyperparameter_tuning/blind_anymal_cfg.py --cfg_class AnymalBlindJobCfg (TERMINAL 2)

I will continue to debug.

[garylvov]

Hi, do you specify the workflow for the tuner run? Thanks for sharing your config, I'm trying those now

[garylvov]

Hi @giulioturrisi

I used the same command to run the Ray server. I am running everything in docker like the new updated documentation. The following command worked fine for me (EDIT: you should update the metric argument to one that is present in the environment, the default doesn't match the env)

./isaaclab.sh -p source/standalone/workflows/ray/tuner.py --cfg_file source/standalone/workflows/ray/hyperparameter_tuning/blind_anymal_cfg.py --cfg_class AnymalBlindJobCfg --run_mode local --workflow source/standalone/workflows/rsl_rl/train.py --num_workers_per_node 4 --metric Train/mean_reward/time # running on my beloved brand new 4 gpu build

I'd also try repulling my branch jic I made changes since the last time you pulled. I'm still waiting for the training runs to finish but nothing looks amiss to me.

EDIT: I'd also update the metric that is being tuned your environment doesn't seem to have the default rewards/time, but the runs seemed to at least start fine for me. I'd check out the logdirs and tensorboard it to see if they really are restarting more often that you'd expect.

[giulioturrisi]

Yes, i modified the field "WORKFLOW" variable to rsl. Just to be sure, now I used the same command you posted (with 1 worker), and still I have the same problem. I'm using conda.

Has you can see, "relative" that is I suppose the time, for me is seconds, for you minutes (I'm using a laptop GPU, 1 worker only). Usually for performing the full training, I took >30 minutes.

Not sure why you have less steps than mine (500 vs 1500) and what they mean. Is it the variable (in my file blind_cfg.py)?:
cfg["hydra_args"]["agent.max_iterations"] = 1500
?

[giulioturrisi]

Yes, the tensorboard is looking good (not so much when used in conjunction with wandb, they seem to interfere - in that case I have flat curve for multiple steps on tensorboard which seems to "run faster than wandb", where the curve are correct - but this is another issue, it is fine turning wand db off in rsl - not anymore fake plateu).

In any case, the process are displayed as TERMINATED in the console, not as ERROR, so it seems fine even if it ends up too quickly. But I'm not sure if the results are correctly taken back

[garylvov]

The process is exiting with code 141 I think there's a simultaneous read/write issue with the logs

[garylvov]

Actually my current theory is that since RSL rl prints out so much it's filling up the pipe lol

[garylvov]

Yeah so in execute_job it needs to flush the subprocess stream, otherwise it crashes prematurely, leading to 141. If we redirect to null it causes 0 prematurely. Working on a fix

[garylvov]

Ok I think I have a fix. Committing it within the hour. Need to verify it on clusters though

[garylvov]

@giulioturrisi can you please give 92bf8e1 a whirl (latest commit on my branch) for your use case in both docker and conda, both with rl games and RSL RL? Will help confirm that everything is now good (or not)

[garylvov]

Bug fix is now on main if you repull should be good!

[giulioturrisi]

Hello! I confirm that everything seems to run normally in my conda environment! I'm currently on the 40th trial in my env with RSL.

Thanks very much for the quick fix!

[glvov-bdai]

Awesome thanks for helping me track down the bug happy training!