🎉MF-MOS achieved a leading IoU of 76.7% on the MOS leaderboard of the SemanticKITTI upon submission, demonstrating the current SOTA performance.
The MOS leaderboard
Video demo
Our pretrained model (best in validation, with the IoU of 76.12%) can be downloaded from Google Drive.
Download SemanticKITTI dataset from SemanticKITTI (including Velodyne point clouds, calibration data and label data).
After downloading the dataset, the residual maps as the input of the model during training need to be generated. Run auto_gen_residual_images.py or auto_gen_residual_images_mp.py(with multiprocess), and check that the path is correct before running.
The structure of one of the folders in the entire dataset is as follows:
DATAROOT
└── sequences
├── 00
│ ├── poses.txt
│ ├── calib.txt
│ ├── times.txt
│ ├── labels
│ ├── residual_images_1
│ ├── residual_images_10
│ ├── residual_images_11
│ ├── residual_images_13
│ ├── residual_images_15
│ ├── residual_images_16
│ ├── residual_images_19
│ ├── residual_images_2
│ ├── residual_images_22
│ ├── residual_images_3
│ ├── residual_images_4
│ ├── residual_images_5
│ ├── residual_images_6
│ ├── residual_images_7
│ ├── residual_images_8
│ ├── residual_images_9
│ └── velodyne
...
If you don't need to do augmentation for residual maps, you just need the folder with num [1, 2, 3, 4, 5, 6, 7, 8].
Our environment: Ubuntu 18.04, CUDA 11.2
Use conda to create the conda environment and activate it:
conda env create -f environment.yml
conda activate mfmosInstall torchsparse which is used in SIEM using the commands:
sudo apt install libsparsehash-dev
pip install --upgrade git+https://github.com/mit-han-lab/[email protected]Check the path in dist_train.sh, and run it to train:
bash script/dist_train.shYou can change the number of GPUs as well as ID to suit your needs.
Once you have completed the first phase of training above, you can continue with SIEM training to get an improved performance.
Check the path in train_siem.sh and run it to train the SIEM (only available on single GPU):
bash script/train_siem.shCheck the path in valid.sh and evaluate.sh.
Then, run them to get the predicted results and IoU in the paper separately:
bash script/valid.sh
# evaluation after validation
bash script/evaluate.shYou can also use our pre-trained model which has been provided above to validate its performance.
Check the path in visualize.sh, and run it to visualize the results in 2D and 3D:
bash script/visualize.shIf -p is empty: only ground truth will be visualized.
If -p set the path of predictions: both ground truth and predictions will be visualized.
Check the path in viz_seqVideo.py, and run it to visualize the entire sequence in the form of a video.
This repo is based on MotionSeg3D and LiDAR-MOS, we are very grateful for their excellent work. Besides, excellent works like 4DMOS[paper, code] and MapMOS[paper, code] have not only demonstrated excellent dynamic object segmentation capabilities on the SemanticKITTI-MOS benchmark but have also exhibited nice generalization abilities on new datasets, which MF-MOS fails to achieve. We appreciate their contributions to MOS and highly recommend people to use their excellent public available code.
