Fake camera stream and use local video file instead

[xuming-shanghai]

Hi Alexey and guys:
Can you help provide the first view video recorded by your drone when your were testing in forest?
It's hard to simulate forest environment in shanghai city.
so, I assume let my drone read your video file(not my camera), then drone thoughts it is in forest.
now I put my drone in outdoor playground, other steps are same as yours except input video steam are faked.
If observed the drone's flight route is along with the road direction in the video, so I think I copied your demo.
Is it feasible about my thought?
Thanks ahead!

[Alexey-Kamenev]

Hello,
That sounds like a good plan. We haven't uploaded the raw drone FPV video, let me check if that can be done quickly.
For now, you can first try playing our test images using image_pub node to make sure your build produces expected results on most basic scenarios. Essentially you need to make sure that if you play, say, "right" image continuously then your drone makes something that vaguely resembles a circle.
Next, you can feed videos from IDSIA Forest Trails dataset using the same image_pub node.
I would recommend doing both steps in simulator first before going outside.
Let us know how it works and don't forget to record and share the videos! :)

[xuming-shanghai]

Let me ensure the drone can position hold with px4low first, today I find a wide and open place to test optical flow. Unfortunately, the drone always drifts from right to left, but altitude looks keeping good, I can see the led is green. Before that, I checked px4flow were connected okay, I hand it with 1 meter distance to ground, the led can turn green.
I have few experience on px4flow, could you help me analysis what happened with it according your experience. Some parameters are set incorrectly?
btw, I have calibrate the sensor with 1m distance.
here is the parameters I referred:

Name value
BFLOW_F_THLD 5000
BFLOW_V_THLD 30
BFLOW_HIST_FIL 0
BFLOW_GYRO_COM 1
BFLOW_LP_FIL 0
BFLOW_W_NEW 0.3
DEBUG 0
GYRO_SENS_DPS 250
GYRO_COMP_THR 0.01
IMAGE_WIDTH 64
IMAGE_HEIGHT 64
IMAGE_L_LIGHT 0
IMAGE_NOISE_C 1
IMAGE_TEST_PAT 0
LENS_FOCAL_LEN 16
POSITION 0
SONAR_FILTERED 0
SONAR_KAL_L1 0.8461
SONAR_KAL_L2 6.2034
SYS_ID 81
SYS_COMP_ID 50
SYS_SENSOR_ID 77
SYS_TYPE 0
SYS_AP_TYPE 1
SYS_SW_VER 13XX
SYS_SEND_STATE 1
USART_2_BAUD 115200
USART_3_BAUD 115200
USB_SEND_VIDEO 1
USB_SEND_FLOW 1
USB_SEND_GYRO 1
USB_SEND_FWD 0
USB_SEND_DEBUG 1
VIDEO_RATE 150
VIDEO_ONLY 0

[Alexey-Kamenev]

Making PX4FLOW work can sometimes be a bit tricky indeed. Please make sure that you read and followed our PX4FLOW instructions. That is, check the firmware, lenses and camera video quality. If everything is set up correctly then PX4FLOW should initialize when you lift the drone above the ground about 20-30 cm (as described here).
It might be also useful to see the output of PX4FLOW module in QGroundControl: connect PX4FLOW to your computer via USB, run QGC and select Widgets -> Analyze menu. Add PX4FlOW topics (I don't remember the names but it should be clear from topic names) and check that x,y coordinates are stable.
If PX4FLOW is configured properly and there is enough texture on the ground then the drone should not drift while flying.

P.S. I'm re-assigning the issue to Nikolai as I'm at the conference now.

[xuming-shanghai]

Thanks, I will try.
btw, image_pub node looks not built in ws.
Can you make changes as below, it works well on my tx1.

diff --git a/ros/scripts/jetson_ros_install.sh b/ros/scripts/jetson_ros_install.sh
index 1d036bc..e0c207d 100755
--- a/ros/scripts/jetson_ros_install.sh
+++ b/ros/scripts/jetson_ros_install.sh
@@ -143,6 +143,7 @@ if [ ! -L "$CATKIN_WS/src/caffe_ros" ]; then
# Create symlinks to catkin workspace.
ln -s $HOME/redtail/ros/packages/caffe_ros $CATKIN_WS/src/
ln -s $HOME/redtail/ros/packages/px4_controller $CATKIN_WS/src/
+ ln -s $HOME/redtail/ros/packages/image_pub $CATKIN_WS/src/
fi

echo "Installing dependencies..."
@@ -162,6 +163,8 @@ echo "Building caffe_ros package..."
catkin_make caffe_ros_node
echo "Building px4_controller package..."
catkin_make px4_controller_node
+echo "Building image_pub package..."
+catkin_make image_pub_node

# Environment setup.
echo "source $CATKIN_WS/devel/setup.bash" >> $HOME/.bashrc

diff --git a/ros/packages/image_pub/CMakeLists.txt b/ros/packages/image_pub/CMakeLists.txt
index a2f31ff..05c3704 100644
--- a/ros/packages/image_pub/CMakeLists.txt
+++ b/ros/packages/image_pub/CMakeLists.txt
@@ -6,6 +6,14 @@ find_package(catkin REQUIRED COMPONENTS
std_msgs
)

+catkin_package(
+### INCLUDE_DIRS include
+ LIBRARIES image_pub_node
+ CATKIN_DEPENDS roscpp std_msgs
+#### DEPENDS system_lib
+)
+
+
###########
## Build ##
###########

[xuming-shanghai]

Hi Alexey, Nikolai
My px4flow is not ready now.
So, I use gazebo simulation on x64 system to simulate image pub and offboard flight control.
The configuration is a bit different with yours, my configuration is real tx1(image pub, mavros, trailnet_dnn, yolo_dnn, px4-controller) + x64 PC(gazebo, qground), that can workaround your nvidia graphics card on PC and also without installing docker.io. I think the principle is the same.
As you mentioned before, when pub a jpg image(redtail/ros/packages/caffe_ros/tests/data/rot_c.jpg) with repeatable mode, the drone circle and circle....
I think it's the reasonable logic, the same frame/image input, the same angle output; so, circle...circle.., but it's not the expected behavior, if the previous one frame was complete same as current one, the dnn network should ignore it, and keep the drone stop and stay in current position. Another test, when I ctrl+c image pub node, there was no image data input, the drone stopped and stay in current position, it looks safe to us.

[Alexey-Kamenev]

Your setup is another good option to test/develop the components. Basically, since all our components are implemented as ROS nodes, you can mix and match workstations, Jetsons and Dockers in any way you like :) We decided to provide and support just 2 scenarios, one based on real Jetson and another - Docker. Such scenarios are relatively easily reproducible and repeatable (especially with Docker).

As for the image_pub node and what to do when the same image is published repeatedly: you can change the code to check for image similarity and do something else (like stopping) in case your metric reaches certain threshold. There is currently no safety mechanisms other than very basic ones like when controller dies, PX4 turns off the offboard mode and switches back to position hold.

[nsmoly]

@xuming-shanghai: Hello, for PX4FLOW floating around -- what lens did you use on PX4FLOW unit? The parameters you posted list: LENS_FOCAL_LEN 16 which is 16mm focal length and it is for the lens that comes default with the unit. If you replaced it with 6mm lens like we suggest, then you need to recompile PX4FLOW firmware after changing the focal length in the settings .h file. It is listed in PX4FLOW instructions.

[nsmoly]

@xuming-shanghai: just to explain what's going on if the focal length is wrong - PX4FLOW cannot estimate velocity properly (due to wrong camera model) and so this leads to overcontrol in Pixhawk. It will stabilize, but it will "float" around (typically side to side)