Add one stage model support (i.e. FairMOT)

[PonyMaY]

Hi Alex! Thanks a lot again for your excellent work again. I want to ask you several questions about the operation of the overall system.

(1) It seems like the whole system has these four blocks: preprocessing, tracking, feature extracting and association according to the master process in mot.step() of app.py.

app.py
if args.mot: mot.step(frame)

Can these four parts to be run in parallel to reduce GPU resource utilization and accelerate the system operation? I have noticed that videoio.py has used threading. So can the similar parallelization be applied to other parts?

(2) Also, in step(self, frame): mot.py,

mot.py
embeddings = self.extractor.postprocess()

Here embeddings can be understood as feature martix. I want to know can it be reused in model processing of different features?

I hope you can understand my questions and give me some guidance in your busy schedule!

[GeekAlexis]

1. Ideally yes, the different stages can be pipelined in parallel. This might cause some delay with live stream but the overall FPS should improve. However, python has GIL so threading does not work for non IO tasks. Also GPU and CPU tasks are already parallelized.
2. Not sure what you mean by reusing the embeddings.

[PonyMaY]

(1) If global parallelization is not performed, can we substitute the tracker model(KLT) seperately like replacing the detect model from SSD to YOLOv4？
(2) I means: Can we reuse the feature vectors from model of detecing and ReID? Can these two model be trained by the same dataset? (eg: reuse the feature vectors both in YOLOv4 and OSNet)
(3) There is another question I want to ask for help: It seems that camera motion compensation failed to work when I coverd the camera entirely. Error kept showing like this:
[Warning]: Camera motion estimation failed.
And the detection and tracking function didn't recover.

[GeekAlexis]

1. Currently, KLT runs in parallel with detection. What do you want to replace KLT with?
2. Yes, refer to one-stage network like FairMOT. However, FairMOT is not optimized (slow). It would be nice if the YOLOv5 version of FairMOT can be converted to TensorRT to work with this repo. Pull requests are welcome!
3. Are you using the latest version? This should be fixed in Questions about homography #144. Tracking won't recover until the camera is uncovered.

[GeekAlexis]

Also, I think three stages can potentially be multithreaded, i.e., detection, feature extraction, and association. But even without Python's GIL, because of the frame skip mechanism, these are not run on every frame and the speedup will be negligible.

[PonyMaY]

Thank you for your prompt and enthusiastic response!
(1) We want to seperate the detection and tracking. Will KLT running in parallel with detecting be better than seperated? For example, we want to use YOLOv5 + JDE (https://arxiv.org/abs/1909.12605v1).
(2) We will try to merge the feature vectors from YOLO into OSNet for training. The reason why we want to reuse the feature vectors is keeping the FPS after we add other functions based on your repo. Do you think this measure will be help?
(3) Thank you! We will use the latest version!

[GeekAlexis]

I need to know your target hardware. Keeping KLT is required if you don’t want to run the detector every frame for FPS.

“Reusing the feature vector” is the same as using JDE or FairMOT. It will help if you can get it working with TensorRT, but some ops may not be supported.

[PonyMaY]

I need to know your target hardware.

We use Jetson NX.

[GeekAlexis]

For Jetson, it's recommended to keep KLT because the detector and feature extractor are too expensive to run at every frame. Unless you can get JDE or FairMOT converted to TensorRT and it runs reasonably fast.

[GeekAlexis]

Marking this as a feature request for one stage model. You should look at FairMOT instead of JDE because it is more accurate. Let me know if you make any progress on conversion to TensorRT. Also related: AlexeyAB/darknet#3042 (comment)

[PonyMaY]

Let me know if you make any progress on conversion to TensorRT.

Hi Alexis! Is the conversion of FairMOT's DLA34 backbone version(ONNX -> TRT) helpful?

[GeekAlexis]

DLA34 is probably slow. Converting their YOLOv5s backbone to TRT is better. You might need YOLOv5 TRT plugins.

[GeekAlexis]

Related: #184

[PonyMaY]

Hi Alexis, sorry for being late to reply you. I have managed to convert FairMOT's YOLOv5 backbone version to TensorRT. Please check it from the download link below.
https://pan.baidu.com/s/14ucwaxZtfoj53Hre1ueJ7A Password:6yct
By the way, the file size of this model is much smaller than the original one, so please let me know whether it is helpful, thank you!

[akashAD98]

@PonyMaY is there any way to use scaled yolo models such as yolov4-mish,csp & pass to tracker like deep sort,FairMOT
Im not getting any direct way to do it, either I need to use TensorFlow or PyTorch weight. & its supporting only for yolov3,yolov4 , not for scaled yolo ( i mean yolov4-mish.weight into yolov4-mish.pb not supporting)

Im planning for passing this weights i.e yolov4x-mish/csp/swish to object tracker. if you have any script please let us know, even fairMOT also fine. Thanks in advance

[GeekAlexis]

@PonyMaY Thanks. I cannot access your link. Can you share your ONNX model and your TensorRT conversion script if possible? Did you use any plugins?

[GeekAlexis]

@akashAD98 FastMOT supports scaled YOLO models. Please open a new issue if you have further questions

[PonyMaY]

@PonyMaY Thanks. I cannot access your link. Can you share your ONNX model and your TensorRT conversion script if possible? Did you use any plugins?

Sorry for making trouble to you to achieve the file.
We referenced this repo https://github.com/wang-xinyu/tensorrtx/tree/master/yolov5 to complete the model conversion.
Now the new link contains both the ONNX and TRT model. Wish they will be helpful. Thanks.
File link: https://drive.google.com/drive/folders/11anLQ06PHN65w6X0IJErs3iOZI7TshLN?usp=sharing
Maybe you should copy the link and access it in a new tab~
By the way, you can check the structure of ONNX model via the tool~ https://netron.app/

[Ritesh1991]

@PonyM and @GeekAlexis have you been integrated yolov5+reid in successfully with fastmot.Sorry for jump in your conversation but I am working on onestage model detection and tracking with yolov5 that's why asking you these question