How to use Stereo DNN with trailnet and px4_controller?

[TritonSailor]

The Stereo DNN work looks very interesting.

How can I use it with the other components of redtail?

Is the intended use case to train trailnet with lidar+photo depth images from stereo dnn? If so, my camera (ZED) can output depth images, would I still use stereo dnn, or can I simply train trailnet in the usual way using depth images?

Will there be further work on the px4_controller released soon? Are there plans to implement waypoints, or velocity setpoints to supplement the currently used rc_override steering?

[nsmoly]

Our work on PX4 for the rover is blocked on PX4 stack properly supporting rovers. We are following up with PX4 team on this right now. Hopefully, if they support offboard for the rovers, then we will update PX4_controller with waypoints. You are right, rc_overrides are not optimal there.
As for stereo DNN - you can host our runtime for it with the provided models as say ROS node. It can consume images from ZED camera (with ZED SDK) and estimate disparity/depth maps. We did this internally and it works.

We were not planning to add this code to GitHub, since thought that stereo runtime is enough. We can do it if we have enough bandwidth due to other projects (at NVIDIA). For now, you can try implementing a ROS node that hosts stereo DNN and contribute it to the project.

A note on performance: our smallest stereo DNN model runs at only 3 FPS on Jetson TX2 (due to use of 3D convolutions). We have a different/less accurate model that can run at 10-30 FPS on TX2. We are thinking to add that binary model to GitHub as well (around June). It was trained specifically for ZED camera. Then the only thing missing will be the ROS node to wrap stereo DNN runtime.

Thanks!