Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update lerobot visualizer to send videos directly to Rerun #476

Open
wants to merge 3 commits into
base: main
Choose a base branch
from

Conversation

jleibs
Copy link

@jleibs jleibs commented Oct 17, 2024

What this does

Rather than using the lerobot loader to decode the video and sending individual images, this now sends the video files directly.

To support this I had to introduce a new video_backend called raw which skips the dataset decoding.

I introduced a new flag --decode-video to maintain the old behavior.

On my desktop this

How it was tested

New functionality:

python lerobot/scripts/visualize_dataset.py  --repo-id lerobot/aloha_static_coffee_new  --episode-index 0

Old functionality:

python lerobot/scripts/visualize_dataset.py  --repo-id lerobot/aloha_static_coffee_new  --episode-index 0 --decode-video

Also verified old image pathway still works on dataset without video:

python lerobot/scripts/visualize_dataset.py     --repo-id lerobot/xarm_lift_medium_image     --episode-index 0

Crude benchmark

This results in a 10x real speedup, a 100x processing-overhead speedup, and a 50x reduction in .rrd file size.

Old pathway:

➜ time python lerobot/scripts/visualize_dataset.py --repo-id lerobot/aloha_static_coffee_new --episode-index 0 --decode-video --save 1 --output-dir tmp

real	0m40.573s
user	7m33.682s
sys	0m53.324s

➜  ls -lh tmp/lerobot_aloha_static_coffee_new_episode_0.rrd 
-rw-r--r-- 1 jleibs jleibs 2.4G Oct 17 17:29 tmp/lerobot_aloha_static_coffee_new_episode_0.rrd

New pathway:

➜ time python lerobot/scripts/visualize_dataset.py --repo-id lerobot/aloha_static_coffee_new --episode-index 0 --save 1 --output-dir tmp

real	0m4.380s
user	0m4.481s
sys	0m2.427s

➜  ls -lh tmp/lerobot_aloha_static_coffee_new_episode_0.rrd 
-rw-r--r-- 1 jleibs jleibs 48M Oct 17 17:31 tmp/lerobot_aloha_static_coffee_new_episode_0.rrd

@Cadene Cadene self-assigned this Oct 18, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants