GeoBEV

[AAAI2025] GeoBEV: Learning Geometric BEV Representation for Multi-view 3D Object Detection

News

• GeoBEV is accepted by AAAI 2025. The paper is available at arxiv.

Main Results

Config	mAP	NDS	Download
GeoBEV-R50-nuImage-CBGS	0.430	0.546	model
GeoBEV-R50-nuImage-CBGS-Longterm	0.479	0.575	model
GeoBEV-R101-nuImage-CBGS-Longterm	0.526	0.615	model

Get Started

1. Please follow these steps to install GeoBEV.

a. Create a conda virtual environment and activate it.

conda create -n geobev python=3.8 -y
conda activate geobev

b. Install PyTorch and torchvision following the official instructions.

pip install torch==1.10.0+cu111 torchvision==0.11.0+cu111 torchaudio==0.10.0 -f https://download.pytorch.org/whl/torch_stable.html

c. Install GeoBEV as mmdet3d.

pip install mmcv-full==1.5.3
pip install mmdet==2.27.0
pip install mmsegmentation==0.25.0
pip install -e .

2. Prepare nuScenes dataset as the below folder structure:

GeoBEV
├── data
│   ├── nuscenes
│   │   ├── lidarseg
│   │   ├── maps
│   │   ├── samples
│   │   ├── samples_instance_mask
│   │   ├── samples_point_label
│   │   ├── sweeps
│   │   ├── v1.0-test
|   |   ├── v1.0-trainval
│   │   ├── geobev-nuscenes_infos_train.pkl
│   │   ├── geobev-nuscenes_infos_val.pkl

a. Download nuScenes 3D detection data HERE and unzip all zip files.

b. The fold samples_instance_mask includes the instance masks of nuScenes images, which are predicted by the HTC model pretrained on nuImages dataset. The prepared data can be downloaded HERE.

c. Create the pkl for GeoBEV by running

python tools/create_data_bevdet.py

d. Download nuScenes-lidarseg annotations HERE and put it under GeoBEV/data/nuscenes/. Create depth and semantic labels from point cloud by running:

python tools/generate_point_label.py

3. Train GeoBEV model on nuScenes:

Download the backbones pretrained on nuImages dataset HERE and put them under GeoBEV/ckpts. Then train the GeoBEV model following:

bash tools/dist_train.sh configs/geobev/geobev-r50-nuimage-cbgs.py 8

4. Evaluate GeoBEV model following:

bash tools/dist_test.sh configs/geobev/geobev-r50-nuimage-cbgs.py work_dirs/geobev-r50-nuimage-cbgs/epoch_20_ema.pth 8 --eval bbox

Acknowledgement

This project is not possible without multiple great open-sourced code bases. We list some notable examples below.

• BEVDet
• BEVDepth
• open-mmlab
• CenterPoint
• Lift-Splat-Shoot

Bibtex

If GeoBEV is helpful for your research, please consider citing the following BibTeX entry.

@article{zhang2024geobev,
  title={Geobev: Learning geometric bev representation for multi-view 3d object detection},
  author={Zhang, Jinqing and Zhang, Yanan and Qi, Yunlong and Fu, Zehua and Liu, Qingjie and Wang, Yunhong},
  journal={arXiv preprint arXiv:2409.01816},
  year={2024}
}