This is the official repository for our ECCV2024 paper CoR-GS: Sparse-View 3D Gaussian Splatting via Co-Regularization.
3D Gaussian Splatting (3DGS) creates a radiance field consisting of 3D Gaussians to represent a scene. With sparse training views, 3DGS easily suffers from overfitting, negatively impacting rendering. This paper introduces a new co-regularization perspective for improving sparse-view 3DGS. When training two 3D Gaussian radiance fields, we observe that the two radiance fields exhibit point disagreement and rendering disagreement that can unsupervisedly predict reconstruction quality, stemming from the randomness of densification implementation. We further quantify the two disagreements and demonstrate the negative correlation between them and accurate reconstruction, which allows us to identify inaccurate reconstruction without accessing ground-truth information. Based on the study, we propose CoR-GS, which identifies and suppresses inaccurate reconstruction based on the two disagreements: (1) Co-pruning considers Gaussians that exhibit high point disagreement in inaccurate positions and prunes them. (2) Pseudo-view co-regularization considers pixels that exhibit high rendering disagreement are inaccurate and suppress the disagreement. Results on LLFF, Mip-NeRF360, DTU, and Blender demonstrate that CoR-GS effectively regularizes the scene geometry, reconstructs the compact representations, and achieves state-of-the-art novel view synthesis quality under sparse training views.
Tested on Ubuntu 18.04, CUDA 11.3, PyTorch 1.12.1
conda env create --file environment.yml
conda activate corgs
pip install gaussian-splatting/submodules/diff-gaussian-rasterization-confidence
pip install gaussian-splatting/submodules/simple-knn
├── /data
├── mipnerf360
├── bicycle
├── bonsai
├── ...
├── nerf_llff_data
├── fern
├── flower
├── ...
├── DTU
├── Rectified
├── scan1
├── scan2
├── ...
├── submission_data
-
Download LLFF from the official download link.
-
run colmap to obtain initial point clouds with limited viewpoints:
python tools/colmap_llff.py
-
Start training and testing:
# for example bash scripts/run_llff.sh ${gpu_id} data/nerf_llff_data/fern output/llff/fern
-
Download MipNeRF-360 from the official download link.
-
run colmap to obtain initial point clouds with limited viewpoints:
python tools/colmap_360.py
-
Start training and testing:
# for example bash scripts/run_360.sh ${gpu_id} data/mipnerf360/bicycle output/mipnerf360/bicycle
-
Download DTU dataset
- Download the DTU dataset "Rectified (123 GB)" from the official website, and extract it.
- Download masks (used for evaluation only) from this link.
-
Organize DTU for few-shot setting:
bash scripts/organize_dtu_dataset.sh $rectified_path -
Format
- Poses: following gaussian-splatting, run
convert.pyto get the poses and the undistorted images by COLMAP. - Render Path: following LLFF to get the
poses_bounds.npyfrom the COLMAP data.
- Poses: following gaussian-splatting, run
-
run colmap to obtain initial point clouds with limited viewpoints:
python tools/colmap_dtu.py
When sparse views are used as input, COLMAP may fail due to poor co-visibility relationships between input images. In our tests, this issue can occur in some scenes within the 3-view DTU and 8-view Blender datasets. In these cases, we use random point clouds for initialization (--rand_pcd).
-
Start training and testing:
python scripts/run_dtu.py
Similar to Gaussian Splatting, our method can read standard COLMAP format datasets. Please customize your sampling rule in scenes/dataset_readers.py, and see how to organize a COLMAP-format dataset from raw RGB images referring to our preprocessing of DTU.
Consider citing as below if you find this repository helpful to your project:
@article{zhang2024cor,
title={CoR-GS: Sparse-View 3D Gaussian Splatting via Co-Regularization},
author={Zhang, Jiawei and Li, Jiahe and Yu, Xiaohan and Huang, Lei and Gu, Lin and Zheng, Jin and Bai, Xiao},
journal={arXiv preprint arXiv:2405.12110},
year={2024}
}
Special thanks to the following awesome projects!