psen_scan_hw_test returns true if test dies by error

[SansoneG]

Commit

397e54b

Steps to reproduce

Execute $TEST_AREA_SCRIPTS_PATH/automatic_acceptance_test_psen_scan.sh 2f9c702

The commit hash is important, as it still includes an error in the scan_compare test. Will be fixed in later commits.

Expected behavior

Script should fail as rostest was not successful

Observed behavior

Script says test was successful (see log)
This line is also interesting (see log)

+ catkin_test_results /var/log/automatic_acceptance_test/2020_11_09_13_36_13_psen_scan_v2
Summary: 0 tests, 0 errors, 0 failures, 0 skipped

Logfiles

[ROSTEST]tearDown[/tmp/catkin_ws/src/psen_scan_v2/test/hw_tests/hwtest_scan_compare.test]
shutting down processing monitor...
[rviz-7] killing on exit
[robot_state_publisher-6] killing on exit
[laser_scanner-5] killing on exit
[ INFO] [1604925581.728655342]: Stop scanner called.
[ INFO] [1604925581.730122335]: Scanner stopped successfully.
... shutting down processing monitor complete
[ROSTEST]rostest teardown /tmp/catkin_ws/src/psen_scan_v2/test/hw_tests/hwtest_scan_compare.test complete
ok
testscan_compare (rostest.runner.RosTest) ... [ROSTEST]setup[/tmp/catkin_ws/src/psen_scan_v2/test/hw_tests/hwtest_scan_compare.test] run_id[a3c20ef8-2288-11eb-bb3e-60f677005c80] starting
started roslaunch server http://psen-scan-test:46199/

SUMMARY
========

PARAMETERS
 * /hztest1/hz: 199.998
 * /hztest1/hzerror: 0.5
 * /hztest1/test_duration: 5.0
 * /hztest1/topic: laser_scanner/scan
 * /laser_scanner/angle_end: 2.39982772149
 * /laser_scanner/angle_start: -2.39982772149
 * /laser_scanner/host_ip: 192.168.0.50
 * /laser_scanner/host_udp_port_control: 55116
 * /laser_scanner/host_udp_port_data: 55115
 * /laser_scanner/prefix: laser_1
 * /laser_scanner/sensor_ip: 192.168.0.10
 * /publishtest/topics: [{'name': 'laser_...
 * /robot_description: <?xml version="1....
 * /rosdistro: melodic
 * /rosversion: 1.14.9
 * /scan_compare/test_duration: 10
 * /scan_compare/testfile: /usr/local/share/...

NODES
  /
    laser_scanner (psen_scan_v2/psen_scan_v2_node)
    robot_state_publisher (robot_state_publisher/robot_state_publisher)
    rviz (rviz/rviz)

[ROSTEST]setup[/tmp/catkin_ws/src/psen_scan_v2/test/hw_tests/hwtest_scan_compare.test] run_id[a3c20ef8-2288-11eb-bb3e-60f677005c80] done
[ROSTEST]Running test [scan_compare]
ROS_MASTER_URI=http://psen-scan-test:33441/
setting /run_id to a3c20ef8-2288-11eb-bb3e-60f677005c80
process[laser_scanner-9]: started with pid [19426]
process[robot_state_publisher-10]: started with pid [19427]
[ INFO] [1604925582.457469593]: Start scanner called.
[ INFO] [1604925582.465030585]: Scanner started successfully.
[ WARN] [1604925582.476872073]: The scanner reports an error: {Device: Master - Error in the External Device Monitoring (EDM1_ERR).}
process[rviz-11]: started with pid [19443]
[ROSTEST]running test scan_compare
process[scan_compare-12]: started with pid [19445]
[==========] Running 1 test from 1 test case.
[----------] Global test environment set-up.
[----------] 1 test from ScanComparisonTests
[ INFO] [1604925582.592030861]: Using testfile /usr/local/share/misc/laser_scan_template.bag
[ RUN      ] ScanComparisonTests.simpleCompare
[ INFO] [1604925583.190644492]: Checking messages for validity. So far looking good.
[ WARN] [1604925583.477632545]: The scanner reports an error: {Device: Master - Error in the External Device Monitoring (EDM1_ERR).}
terminate called after throwing an instance of 'std::bad_alloc'
  what():  std::bad_alloc
[scan_compare-12] process has died [pid 19445, exit code -6, cmd /tmp/catkin_ws/devel/lib/psen_scan_v2/hwtest_scan_compare --gtest_output=xml:/var/log/automatic_acceptance_test/2020_11_09_13_36_13_psen_scan_v2/psen_scan_v2/rosunit-scan_compare.xml --text __name:=scan_compare __log:=/home/gsansone/.ros/log/a3c20ef8-2288-11eb-bb3e-60f677005c80/scan_compare-12.log].
log file: /home/gsansone/.ros/log/a3c20ef8-2288-11eb-bb3e-60f677005c80/scan_compare-12*.log
[ WARN] [1604925584.480084811]: The scanner reports an error: {Device: Master - Error in the External Device Monitoring (EDM1_ERR).}
[ROSTEST]test [scan_compare] finished
[ROSTEST][ROSTEST] test [scan_compare] done
[ROSTEST]tearDown[/tmp/catkin_ws/src/psen_scan_v2/test/hw_tests/hwtest_scan_compare.test]
shutting down processing monitor...
[rviz-11] killing on exit
[robot_state_publisher-10] killing on exit
[laser_scanner-9] killing on exit
[ INFO] [1604925584.557382133]: Stop scanner called.
[ INFO] [1604925584.559318161]: Scanner stopped successfully.
... shutting down processing monitor complete
[ROSTEST]rostest teardown /tmp/catkin_ws/src/psen_scan_v2/test/hw_tests/hwtest_scan_compare.test complete
ok

----------------------------------------------------------------------
Ran 3 tests in 12.476s

OK

WARNING: overall test result is not accurate when --text is enabled
rostest log file is in /home/gsansone/.ros/log/rostest-psen-scan-test-19256.log
+ log_test_results
+ catkin_test_results /var/log/automatic_acceptance_test/2020_11_09_13_36_13_psen_scan_v2
Summary: 0 tests, 0 errors, 0 failures, 0 skipped
execute test done
Test successful

[jschleicher]

Maybe this is due to rostest --text?

WARNING: overall test result is not accurate when --text is enabled

[martiniil]

Maybe this is due to rostest --text?

WARNING: overall test result is not accurate when --text is enabled

yes

[agutenkunst]

Seems the test crashes

[ WARN] [1604925583.477632545]: The scanner reports an error: {Device: Master - Error in the External Device Monitoring (EDM1_ERR).}
terminate called after throwing an instance of 'std::bad_alloc'
  what():  std::bad_alloc

no result xml get generated -> catkin_test_results return 0 since there are no failed tests.