This repository contains the code for the paper "Language-Embedded Gaussian Splats (LEGS): Incrementally Building Room-Scale Representations with a Mobile Robot".
Model-based diffusion (MBD) is a novel diffusion-based trajectory optimization framework that employs a dynamics model to approximate the score function. MBD outperforms existing methods (including RL) in terms of sample efficiency and generalization.
Language Embedded Gaussian Splats follows the integration guidelines described here for custom methods within Nerfstudio.
To learn more about the code we use to interface with the robot and collect image poses, see this repo here, which outlines our ROS2 interface.
Follow these instructions up to and including "tinycudann" to install dependencies.
If you'll be using ROS messages do not use a conda environment and enter the dependency install commands below instead (ROS and conda don't play well together)
pip install torch==2.0.1+cu118 torchvision==0.15.2+cu118 --extra-index-url https://download.pytorch.org/whl/cu118
pip install ninja git+https://github.com/NVlabs/tiny-cuda-nn/#subdirectory=bindings/torch
git clone https://github.com/BerkeleyAutomation/L3GS
cd L3GS/l3gs/
python -m pip install -e .
ns-install-cli
Run ns-train -h: you should see a list of "subcommands" with lllegos and llgs included among them.
- Launch training with
ns-train l3gsand start publishing an imagepose topic or playing an imagepose ROS bag. - Connect to the viewer by forwarding the viewer port (we use VSCode to do this), and click the link to
viewer.nerf.studioprovided in the output of the train script
If you find this useful, please cite the paper!
@article{yu2024language,
author = {Yu, Justin and Hari, Kush and Srinivas, Kishore and El-Refai, Karim and Rashid, Adam and Kim, Chung Min and Kerr, Justin and Cheng, Richard and Irshad, Muhammad Zubair and Balakrishna, Ashwin and Kollar, Thomas and Goldberg, Ken},
title = {Language-Embedded Gaussian Splats (LEGS): Incrementally Building Room-Scale Representations with a Mobile Robot},
booktitle = {International Conference on Intelligent Robots and Systems (IROS)},
year = {2024},
}
