docs(image_projection_based_fusion): add algorithm description of sen…

…sor synchronization (autowarefoundation#2705)

* ci: add sync-upstream.yaml (#4)

Signed-off-by: Kenji Miyake <[email protected]>

* ci(sync-upstream): update settings (autowarefoundation#19)

Signed-off-by: Kenji Miyake <[email protected]>

* chore: sync files (autowarefoundation#629)

* chore: sync files

Signed-off-by: GitHub <[email protected]>

* ci(pre-commit): autofix

Co-authored-by: kenji-miyake <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* chore: sync files (autowarefoundation#637)

Signed-off-by: GitHub <[email protected]>

Co-authored-by: kenji-miyake <[email protected]>

* fix(dummy_diag_publisher): use anon to make unique node name instead of diag name (autowarefoundation#639)

* chore: sync files (autowarefoundation#648)

* chore: sync files

Signed-off-by: GitHub <[email protected]>

* Revert "chore: sync files"

This reverts commit b24f530.

Signed-off-by: Kenji Miyake <[email protected]>

* sync codecov.yaml

Signed-off-by: Kenji Miyake <[email protected]>

Co-authored-by: kenji-miyake <[email protected]>
Co-authored-by: Kenji Miyake <[email protected]>

* fix(autoware_state_panel): fix message type for /api/autoware/get/engage (autowarefoundation#666)

* fix(autoware_state_panel): fix message type for /api/autoware/get/engage

Signed-off-by: h-ohta <[email protected]>

* ci(pre-commit): autofix

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* fix(behavior_velocity): avoid insert same point on trajectory utils (autowarefoundation#834)

Signed-off-by: tanaka3 <[email protected]>

* refactor(behavior_velocity_planner): simplify CMakeLists.txt (autowarefoundation#855)

Signed-off-by: Kenji Miyake <[email protected]>

* fix(system_monitor): fix build error on tegra platform (autowarefoundation#869)

* fix(system_monitor): fix build error on tegra platform

Signed-off-by: Shark Liu <[email protected]>

* ci(pre-commit): autofix

* Update system/system_monitor/src/gpu_monitor/tegra_gpu_monitor.cpp

Co-authored-by: Shark Liu <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daisuke Nishimatsu <[email protected]>

* feat(ad_service_state_monitor): limit odometry buffer size (autowarefoundation#514)

* feat(ad_service_state_monitor): limit odometry buffer size 40

Signed-off-by: Takayuki Murooka <[email protected]>

* Update system/ad_service_state_monitor/src/ad_service_state_monitor_node/ad_service_state_monitor_node.cpp

Co-authored-by: Kenji Miyake <[email protected]>

* Update ad_service_state_monitor_node.cpp

Co-authored-by: Kenji Miyake <[email protected]>

* docs: fix 404 error caused by typo in url (autowarefoundation#871)

* docs: fix 404 error caused by typo in url

Signed-off-by: Shin-kyoto <[email protected]>

* docs: fix typo in url for yolov4

Signed-off-by: Shin-kyoto <[email protected]>

* fix(image_projection_based_fusion): set imagebuffersize (autowarefoundation#820)

* fix: set imagebuffersize configured

Signed-off-by: suchang <[email protected]>

* ci(pre-commit): autofix

Co-authored-by: suchang <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* chore(avoidance_module): fix spell check (autowarefoundation#732)

Signed-off-by: Muhammad Zulfaqar Azmi <[email protected]>

* feat: isolate gtests in all packages (autowarefoundation#693)

Signed-off-by: Maxime CLEMENT <[email protected]>

* docs(virtual traffic light): add documentation (autowarefoundation#245)

* doc(behavior_velocity): add graph and fix link

* doc(behavior_velocity): update virtual traffic light doc

Signed-off-by: tanaka3 <[email protected]>

* doc(behavior_velocity): minor fix

Signed-off-by: tanaka3 <[email protected]>

* doc : mediate to coordinate

Signed-off-by: tanaka3 <[email protected]>

* doc: minor update

Signed-off-by: tanaka3 <[email protected]>

* doc: fix pre-commit

* doc: update docs

Signed-off-by: tanaka3 <[email protected]>

* apply suggestion

* doc: to intersection-coordination

Co-authored-by: Kenji Miyake <[email protected]>

* feat(surround_obstacle_checker): separate surround_obstacle_checker from hierarchical planning flow (autowarefoundation#830)

* fix(surroud_obstacle_checker): use alias

Signed-off-by: satoshi-ota <[email protected]>

* feat(surround_obstacle_checker): use velocity limit

Signed-off-by: satoshi-ota <[email protected]>

* chore(surround_obstacle_checker): rename publisher, subscriber and callback functions

Signed-off-by: satoshi-ota <[email protected]>

* refactor(surround_obstacle_checker): use parameter struct

Signed-off-by: satoshi-ota <[email protected]>

* fix(surround_obstacle_checker): use alias

Signed-off-by: satoshi-ota <[email protected]>

* refactor(surround_obstacle_checker): cleanup member functions

Signed-off-by: satoshi-ota <[email protected]>

* refactor(surround_obstacle_checker): cleanup polygon handling

Signed-off-by: satoshi-ota <[email protected]>

* refactor(surround_obstacle_checker): use marker helper

Signed-off-by: satoshi-ota <[email protected]>

* feat(planning_launch): separate surround_obstacle_checker from hierarchical motion planning flow

Signed-off-by: satoshi-ota <[email protected]>

* feat: remove deprecated package in prediction launch (autowarefoundation#875)

Signed-off-by: Yukihiro Saito <[email protected]>

* fix(surround_obstacle_checker): fix ego footprint polygon (autowarefoundation#877)

Signed-off-by: satoshi-ota <[email protected]>

* fix: update nvinfer api (autowarefoundation#863)

* fix(lidar_centerpoint): update nvinfer api

Signed-off-by: Daisuke Nishimatsu <[email protected]>

* fix(tensorrt_yolo): update nvinfer api

Signed-off-by: Daisuke Nishimatsu <[email protected]>

* fix(lidar_apollo_instance_segmentation): update nvinfer api

Signed-off-by: Daisuke Nishimatsu <[email protected]>

* fix(traffic_light_classifier): update nvinfer api

Signed-off-by: Daisuke Nishimatsu <[email protected]>

* fix(traffic_light_ssd_fine_detector): update nvinfer api

Signed-off-by: Daisuke Nishimatsu <[email protected]>

* pre-commit run

Signed-off-by: Daisuke Nishimatsu <[email protected]>

* fix(avoidance_module): ignore object instead of creating zero shift (autowarefoundation#731)

* fix: ignore object instead of creating zero shift

instead of creating zero shift point, the object will be ignored.
no behavior changes should be observed.

Signed-off-by: Muhammad Zulfaqar Azmi <[email protected]>

* refactor: sync continue with upstream

Signed-off-by: Muhammad Zulfaqar Azmi <[email protected]>

* fix: fix debug message for insufficient lateral margin

Signed-off-by: Muhammad Zulfaqar Azmi <[email protected]>

* fix(motion_velocity_smoother): curve deceleration not working with a specific parameter set (autowarefoundation#738)

Signed-off-by: Takamasa Horibe <[email protected]>

* test(autoware_testing): fix smoke_test (autowarefoundation#479)

* fix(autoware_testing): fix smoke_test

Signed-off-by: Keisuke Shima <[email protected]>

* restore smoke_test for trajectory_follower_nodes

Signed-off-by: Keisuke Shima <[email protected]>

* add support multiple parameter files

Signed-off-by: Keisuke Shima <[email protected]>

* ci(pre-commit): autofix

* minor fix

Signed-off-by: Keisuke Shima <[email protected]>

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* feat(rviz_plugins): add velocity limit to autoware state panel (autowarefoundation#879)

* feat(rviz_plugins): add velocity limit to autoware state panel

Signed-off-by: tanaka3 <[email protected]>

* chore(rviz_plugin): change ms to kmh

Signed-off-by: tanaka3 <[email protected]>

* fix(dummy_perception_publisher): publish multiple layers of pointcloud (autowarefoundation#882)

* fix: single -> multiple layers pointcloud

Signed-off-by: Hirokazu Ishida <[email protected]>

* refactor: share common among different pcloud creators

Signed-off-by: Hirokazu Ishida <[email protected]>

* feat(vehicle_info_util): add max_steer_angle (autowarefoundation#740)

* feat(vehicle_info_util): add max_steer_angle

Signed-off-by: Takayuki Murooka <[email protected]>

* applied pre-commit

Signed-off-by: Takayuki Murooka <[email protected]>

* Added max_steer_angle in test config

Signed-off-by: Takayuki Murooka <[email protected]>

Co-authored-by: Tomoya Kimura <[email protected]>

* ci(deploy-docs): remove mdx_unimoji (autowarefoundation#883)

Signed-off-by: Shumpei Wakabayashi <[email protected]>

* chore: sync files (autowarefoundation#884)

Signed-off-by: GitHub <[email protected]>

Co-authored-by: kenji-miyake <[email protected]>

* fix(lidar_centerpoint): fix google drive url to avoid 404 (autowarefoundation#889)

* fix(lidar_centerpoint): fix google drive url to avoid 404

* Update CMakeLists.txt

Co-authored-by: Kenji Miyake <[email protected]>

* chore: fix typos (autowarefoundation#886)

Signed-off-by: badai-nguyen <[email protected]>

* feat(state_rviz_plugin): add GateMode and PathChangeApproval Button (autowarefoundation#894)

* feat(state_rviz_plugin): add GateMode and PathChangeApproval Button

* ci(pre-commit): autofix

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* feat(map_tf_generator): accelerate the 'viewer' coordinate calculation (autowarefoundation#890)

* add random point sampling function to quickly calculate the 'viewer' coordinate

Signed-off-by: IshitaTakeshi <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Kenji Miyake <[email protected]>

* docs(obstacle_stop_planner): update documentation (autowarefoundation#880)

Signed-off-by: satoshi-ota <[email protected]>

* docs(tier4_traffic_light_rviz_plugin): update documentation (autowarefoundation#905)

Signed-off-by: satoshi-ota <[email protected]>

* fix(accel_brake_map_calibrator): rviz panel type (autowarefoundation#895)

* fixed panel type

Signed-off-by: Mamoru Sobue <[email protected]>

* modified instruction for rosbag replay case

Signed-off-by: Mamoru Sobue <[email protected]>

* modified update_map_dir service name

Signed-off-by: Mamoru Sobue <[email protected]>

* fix(behavior velocity planner): skipping emplace back stop reason if it is empty (autowarefoundation#898)

* skipping emplace back stop reason if it is empty

Signed-off-by: TakumiKozaka-T4 <[email protected]>

* add braces

Signed-off-by: TakumiKozaka-T4 <[email protected]>

* ci(pre-commit): autofix

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Takagi, Isamu <[email protected]>

* feat(behavior_path_planner): weakened noise filtering of drivable area (autowarefoundation#838)

* feat(behavior_path_planner): Weakened noise filtering of drivable area

Signed-off-by: Takayuki Murooka <[email protected]>

* fix lanelet's longitudinal disconnection

Signed-off-by: Takayuki Murooka <[email protected]>

* add comments of erode/dilate process

Signed-off-by: Takayuki Murooka <[email protected]>

* ci: add slack-send.yaml (autowarefoundation#133)

Signed-off-by: Kenji Miyake <[email protected]>

Signed-off-by: Kenji Miyake <[email protected]>

* add: update readme

Signed-off-by: tzhong518 <[email protected]>

* ci(pre-commit): autofix

* fix: remove unneeded file

Signed-off-by: tzhong518 <[email protected]>

* fix: table format

Signed-off-by: tzhong518 <[email protected]>

* ci(pre-commit): autofix

* Update perception/image_projection_based_fusion/README.md

Co-authored-by: Shunsuke Miura <[email protected]>

* Update README.md

* Update README.md

* Update README.md

---------

Signed-off-by: Kenji Miyake <[email protected]>
Signed-off-by: tanaka3 <[email protected]>
Signed-off-by: Shin-kyoto <[email protected]>
Signed-off-by: Muhammad Zulfaqar Azmi <[email protected]>
Signed-off-by: Maxime CLEMENT <[email protected]>
Signed-off-by: satoshi-ota <[email protected]>
Signed-off-by: Yukihiro Saito <[email protected]>
Signed-off-by: Daisuke Nishimatsu <[email protected]>
Signed-off-by: Takamasa Horibe <[email protected]>
Signed-off-by: Hirokazu Ishida <[email protected]>
Signed-off-by: Shumpei Wakabayashi <[email protected]>
Signed-off-by: badai-nguyen <[email protected]>
Signed-off-by: IshitaTakeshi <[email protected]>
Signed-off-by: Mamoru Sobue <[email protected]>
Signed-off-by: Takayuki Murooka <[email protected]>
Signed-off-by: tzhong518 <[email protected]>
Co-authored-by: Kenji Miyake <[email protected]>
Co-authored-by: tier4-autoware-public-bot[bot] <98652886+tier4-autoware-public-bot[bot]@users.noreply.github.com>
Co-authored-by: awf-autoware-bot[bot] <94889083+awf-autoware-bot[bot]@users.noreply.github.com>
Co-authored-by: kenji-miyake <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Hiroki OTA <[email protected]>
Co-authored-by: Yuma Nihei <[email protected]>
Co-authored-by: Kenji Miyake <[email protected]>
Co-authored-by: taikitanaka3 <[email protected]>
Co-authored-by: Shark <[email protected]>
Co-authored-by: Shark Liu <[email protected]>
Co-authored-by: Daisuke Nishimatsu <[email protected]>
Co-authored-by: Takayuki Murooka <[email protected]>
Co-authored-by: Shintaro Tomie <[email protected]>
Co-authored-by: storrrrrrrrm <[email protected]>
Co-authored-by: suchang <[email protected]>
Co-authored-by: Zulfaqar Azmi <[email protected]>
Co-authored-by: Maxime CLEMENT <[email protected]>
Co-authored-by: Satoshi OTA <[email protected]>
Co-authored-by: Yukihiro Saito <[email protected]>
Co-authored-by: Takamasa Horibe <[email protected]>
Co-authored-by: Keisuke Shima <[email protected]>
Co-authored-by: Hirokazu Ishida <[email protected]>
Co-authored-by: Tomoya Kimura <[email protected]>
Co-authored-by: Shumpei Wakabayashi <[email protected]>
Co-authored-by: badai nguyen <[email protected]>
Co-authored-by: Takeshi Ishita <[email protected]>
Co-authored-by: Mamoru Sobue <[email protected]>
Co-authored-by: TakumiKozaka-T4 <[email protected]>
Co-authored-by: Takagi, Isamu <[email protected]>
Co-authored-by: Yohei Mishina <[email protected]>
Co-authored-by: naokimatsunawa <[email protected]>
Co-authored-by: Tomohito ANDO <[email protected]>
Co-authored-by: Shunsuke Miura <[email protected]>

perception/image_projection_based_fusion/README.md

@@ -6,9 +6,55 @@ The `image_projection_based_fusion` is a package to fuse detected obstacles (bou
 
 ## Inner-workings / Algorithms
 
-Detail description of each fusion's algorithm is in the following links.
+### Sync Algorithm
+
+#### matching
+
+The offset between each camera and the lidar is set according to their shutter timing.
+After applying the offset to the timestamp, if the interval between the timestamp of pointcloud topic and the roi message is less than the match threshold, the two messages are matched.
+
+![roi_sync_image1](./docs/images/roi_sync_1.png)
+
+current default value at autoware.universe for TIER IV Robotaxi are: - input_offset_ms: [61.67, 111.67, 45.0, 28.33, 78.33, 95.0] - match_threshold_ms: 30.0
+
+#### fusion and timer
+
+![roi_sync_image2](./docs/images/roi_sync_2.png)
+
+The subscription status of the message is signed with 'O'.
+1.if a pointcloud message is subscribed under the below condition:
+| | pointcloud | roi msg 1 | roi msg 2 | roi msg 3 |
+| :-----------------: | :--------: | :-------: | :-------: | :-------: |
+| subscription status | | O | O | O |
+
+If the roi msgs can be matched, fuse them and postprocess the pointcloud message.
+Otherwise, fuse the matched roi msgs and cache the pointcloud.
+2.if a pointcloud message is subscribed under the below condition:
+| | pointcloud | roi msg 1 | roi msg 2 | roi msg 3 |
+| :-----------------: | :--------: | :-------: | :-------: | :-------: |
+| subscription status | | O | O | |
+
+if the roi msgs can be matched, fuse them and cache the pointcloud.
+3.if a pointcloud message is subscribed under the below condition:
+| | pointcloud | roi msg 1 | roi msg 2 | roi msg 3 |
+| :-----------------: | :--------: | :-------: | :-------: | :-------: |
+| subscription status | O | O | O | |
+
+If the roi msg 3 is subscribed before the next pointcloud messge coming or timeout, fuse it if matched, otherwise wait for the next roi msg 3.
+If the roi msg 3 is not subscribed before the next pointcloud messge coming or timeout, postprocess the pointcloud messege as it is.
+
+The timeout threshold should be set according to the postprocessing time.
+E.g, if the postprocessing time is around 50ms, the timeout threshold should be set smaller than 50ms, so that the whole processing time could be less than 100ms.
+current default value at autoware.universe for XX1: - timeout_ms: 50.0
+
+#### Known Limits
+
+The rclcpp::TimerBase timer could not break a for loop, therefore even if time is out when fusing a roi msg at the middle, the program will run until all msgs are fused.
+
+### Detail description of each fusion's algorithm is in the following links
 
 | Fusion Name                | Description                                                                                     | Detail                                       |
 | -------------------------- | ----------------------------------------------------------------------------------------------- | -------------------------------------------- |
 | roi_cluster_fusion         | Overwrite a classification label of clusters by that of ROIs from a 2D object detector.         | [link](./docs/roi-cluster-fusion.md)         |
 | roi_detected_object_fusion | Overwrite a classification label of detected objects by that of ROIs from a 2D object detector. | [link](./docs/roi-detected-object-fusion.md) |
+| pointpainting_fusion       | Paint the point cloud with the ROIs from a 2D object detector and feed to a 3D object detector. | [link](./docs/pointpainting-fusion.md)       |