Annotation of NuScenes Dataset in Dev2.0 Branch

[zw615]

Hi, I accidentally notice that there are both 'ann_infos' here and 'ann_info' here the get_data_info of nuScenes dataset code. The names of these two variables are so close, that I do not know their difference.

It seems 'ann_infos' is created first in create_data_bevdet.py, while 'ann_info' is created with 'gt_boxes', 'gt_names', and 'gt_velocity', which are first created in nuscenes_converter.py. If my understanding is correct, both data preparation calls nuscenes_converter.create_nuscenes_infos, but the new create_data_bevdet.py add a 'ann_infos' attribute to the info pickle file afterwards.

I wonder what is the difference between the ground-truth bboxes information (pos, size, rot, velo, etc...) in 'ann_infos' and 'ann_info'? Like they are based on different coordinate system? Or rather, is there any differentce? Can they be used interchangeably?
BTW, I see that 'ann_infos' is introduced with the new Dev2.0 branch, and 'ann_info' is used in the previous master branch.

[zw615]

Further investigation shows in loading.py, 'ann_infos' is in LoadAnnotationsBEVDepth like here; 'ann_info' is used LoadAnnotations3D like here.

Does this mean, even though BEVDet-depth is using centerpoint head, the new bevdet in Dev2.0 Branch is actually predicting bbox format different from the original CenterPoint model? Assuming they are using the same bevdetv2-nuscenes_infos_train.pkl? As the former uses LoadAnnotationsBEVDepth, but the latter uses LoadAnnotations3D?

[zw615]

Interestingly, I find that using dev2.0 branch code and bevdetv2-nuscenes_infos_train.pkl cannot train a centerpoint successfully. The training loss decreases normally, but the prediction is a complete mess. The mAP and NDS are both zero all the time. My guess is bevdet now uses 'ann_infos' in LoadAnnotationsBEVDepth, and centerpoint uses 'ann_info' in LoadAnnotations3D, while these two annotations are based on different coordinate systems. Is my guess right?

[zw615]

After a more careful inspection of the NuScenes dataset code, I think the reason is that now the detection results are supposed to be in the camera ego reference frame, while the LiDAR-based detectors like CenterPoint use LiDAR reference frame. I will try to convert the reference frame manually to suit the new NuScenes dataset code, and see if it works.

[MaRongbo]

the detection results are supposed to be in the camera ego reference frame, while the LiDAR-based detectors like CenterPoint use LiDAR reference frame.

The nusences's ego coordinate is same with mm3d's lidar coordinate. BEVdet2.0 transformed the annos to LiDARInstance3DBoxes is here . I think the reason is DEVdet2.0 modify function _format_bbox in nusenes_dataset.py, it change the predict result from ego to global coordinate, so your mAP and NDS is 0 all the time.

[MaRongbo]

but I have a strange problem, my own dataset's ego reference is same with nuScenes's ego reference , when I train my own dataset use BEVDet2.0, the predict result is also mess, it seems rotated 90 degrees. Anyone has some idea?

[WangJian981002]

Interestingly, I find that using dev2.0 branch code and bevdetv2-nuscenes_infos_train.pkl cannot train a centerpoint successfully. The training loss decreases normally, but the prediction is a complete mess. The mAP and NDS are both zero all the time. My guess is bevdet now uses 'ann_infos' in LoadAnnotationsBEVDepth, and centerpoint uses 'ann_info' in LoadAnnotations3D, while these two annotations are based on different coordinate systems. Is my guess right?

Hi, I encountered the following problems in the process of using dev2.0 branch to train centerpoint (I have successfully trained configs/bevdet/bevdet-r50-cbgs.py before this)

the training code （model）：
bash ./tools/dist_train.sh configs/centerpoint/centerpoint_01voxel_second_secfpn_dcn_4x8_cyclic_20e_nus.py 4

and the error:
Original Traceback (most recent call last): File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/_utils/worker.py", line 287, in _worker_loop data = fetcher.fetch(index) File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 49, in fetch data = [self.dataset[idx] for idx in possibly_batched_index] File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 49, in <listcomp> data = [self.dataset[idx] for idx in possibly_batched_index] File "/home/wj/BEVDet-dev2.0/mmdet3d/datasets/dataset_wrappers.py", line 68, in __getitem__ return self.dataset[ori_idx] File "/home/wj/BEVDet-dev2.0/mmdet3d/datasets/custom_3d.py", line 435, in __getitem__ data = self.prepare_train_data(idx) File "/home/wj/BEVDet-dev2.0/mmdet3d/datasets/custom_3d.py", line 229, in prepare_train_data example = self.pipeline(input_dict) File "/home/wj/BEVDet-dev2.0/mmdet3d/datasets/pipelines/compose.py", line 49, in __call__ data = t(data) File "/home/wj/BEVDet-dev2.0/mmdet3d/datasets/pipelines/loading.py", line 664, in __call__ results = self._load_bboxes_3d(results) File "/home/wj/BEVDet-dev2.0/mmdet3d/datasets/pipelines/loading.py", line 559, in _load_bboxes_3d results['gt_bboxes_3d'] = results['ann_info']['gt_bboxes_3d'] KeyError: 'ann_info'
it seems that the "ann_info" is missing when loading the 3d_box_annos, howeve, when I use mmdetection3d-1.0.0rc4 to train the same model, there will be no error. Do you know what is the reason? I am new to mmdet3d and hope to get your reply, thank u!

[lilkeker]

在仔细检查 NuScenes 数据集代码后，我认为原因是现在检测结果应该在相机自我参考框架中，而基于 LiDAR 的检测器（如 CenterPoint）使用 LiDAR 参考框架。我将尝试手动转换参考帧以适应新的 NuScenes 数据集代码，并查看它是否有效。

Hello, have you solved this problem? I evaled my pre-trained transfusion-lidar under bevdet-dev2.0 and encountered the problem that both map and nds are 0. Now I don’t know how to solve it. I can only use it in bevdet -dev2.0 retrain a model, I am thinking of implementing bevfusion on the basis of bevdet

[123mutouren-lhq]

but I have a strange problem, my own dataset's ego reference is same with nuScenes's ego reference , when I train my own dataset use BEVDet2.0, the predict result is also mess, it seems rotated 90 degrees. Anyone has some idea?

hello,do you deal with this problem? I meet too