autoware test

[storrrrrrrrm]

how is autoware tested now? i noticed that there are some gtest programs, but it seems that there is no gtest programms excuted in a github action.

Do you test on a specific bag to ensure that new code changes don't affect existing functionality?

[kenji-miyake]

@storrrrrrrrm Regarding gtest (or launch_testing), it's executed through colcon test.
https://github.com/autowarefoundation/autoware.universe/blob/49d1764a5cb02924910cc5fab0f055cc337ca8ac/.github/workflows/build-and-test.yaml#L36

But there are not so many unit tests in autoware.universe now.
Instead, TIER IV utilizes scenario testing by https://github.com/tier4/scenario_simulator_v2.
Although it's currently only targeting Planning components, we're preparing to extend the support using Rosbags or a Digital-twin simulator.

And in the future, we'll refactor the code of autoware.universe and put it into autoware.core, and there will be much more unit tests.
For that, we are discussing the interface at the Architecture Meetings now.

[maxime-clem]

If you want a test with multiple nodes running in parallel and communicating with each other, this corresponds to an integration test and should use launch_testing as isamu-takagi said.

If you only want to test a single node, you can manually publish the input topics and check the output topic. You can find examples of such test in the trajectory_follower_nodes package: https://github.com/autowarefoundation/autoware.universe/blob/main/control/trajectory_follower_nodes/test/
This requires creating a "fake" node to publish and subscribe to some topics. A convenience library is available to help create this fake node: https://github.com/autowarefoundation/autoware.universe/blob/main/common/fake_test_node/design/fake_test_node-design.md

[storrrrrrrrm]

@isamu-takagi @maxime-clem can u tell me how to start this test? i use

python3 ./install/default_ad_api/share/default_ad_api/test/main.test.py

it shows nothing

and i check https://github.com/autowarefoundation/autoware.universe/runs/6176222765?check_suite_focus=true wanna to find how it is tested,but find nothing

[isamu-takagi]

@storrrrrrrrm For launch_testing, use launch_test command.
launch_test ./install/default_ad_api/share/default_ad_api/test/main.test.py

[storrrrrrrrm]

let me summarize my question

1. if i modify the code of package default_ad_api and create a PR, would cmd launch_test ./install/default_ad_api/share/default_ad_api/test/main.test.py triggerd in ci process?
2. as far as i understand,through main.test.py, two .py files are executed,
   1. interface_version.py
      this is test script to test InterfaceVersionNode
   2. default_ad_api.launch.py
      InterfaceVersionNode and web_server.py are called by default_ad_api.launch.py, these two nodes(InterfaceVersionNode and web_server.py) are 'multi multiple nodes running in parallel and communicating with each other'
      am i right?

what i want to do is:
write a test node to test multi perception nodes(crop_box_filter/ground_filter/voxel_grid_filter/euclidean_cluster/tensorrt_yolo/rois_cluster_fusion/..) for autoware,so problems can be found in time
after codes related with these nodes modified

my initial idea was writing a script to start all perception nodes in background, writing a gtest program to publish fake lidar data and image data,and check results from rois_cluster_fusion. the only problem is how to integrate it into the existing autoware CI process

if using launch_testing as u mentioned, i need to write three .py files

1. main.test.py
   call start_nodes.py and result_check.py
2. start_nodes.py,like default_ad_api.launch.py
   start all perception nodes. is it technically feasible to start all these codes correctly with specific params like we use 'ros2 launch autoware_launch perception.launch.xml'
3. result_check.py,like interface_version.py
   construct fake lidar data,recv result from rois_cluster_fusion node,check the result

it looks too complex and hard to achieve.

[maxime-clem]

Your plan if using launch_testing seems good.
Constructing fake lidar data does sound difficult but this can be replaced by using real data (e.g., played back from a bag).
There is also a package in .Auto which constructs fake 2D lidar data and might be useful to you: https://gitlab.com/autowarefoundation/autoware.auto/AutowareAuto/-/blob/master/src/perception/tracking_test_framework/design/tracking_test_framework-design.md