Unable to start the Autonomous Driving mode ("Auto") in a multi launch-file configuration

[rt-2pm2]

I am new to Autoware, and I am trying running a demo (based on the Planning Simulation one) where the Autoware framework nodes are distributed among 2 hosts on the same network, precisely:

• a workstation running all the components, except for the control node, and
• an embedded board running the control node only.

Description of the problem:
After having placed the vehicle and generated the trajectory to a target point in the map, the "Auto" button which should enable the automatic trajectory following is grayed out and the vehicle remains in a "stop" condition.

Details

I am running the containerized version of Autoware on both the hosts: the workstation is using the nvidia/cuda enabled docker image, whilst the embedded board is running the container based on the docker image without nvidia/cuda.

Inspecting the launch files, I saw that the control part is managed by the "launch_control" argument in the autoware.launch.xml, which leads to the inclusion of the tier4_control_component.launch.xml.
For this reason, on the embedded board, I prepared a launch file which supplies that component only, precisely:

<?xml version="1.0" encoding="UTF-8"?>
<launch>
	<arg name="vehicle_model" default="sample_vehicle"/>
	<arg name="vehicle_id" default="$(env VEHICLE_ID default)" />
	<arg name="check_external_emergency_heartbeat" default="false" />
	<include file="$(find-pkg-share autoware_launch)/launch/components/tier4_control_component.launch.xml"/>
</launch>

On the Workstation, the core part launching the Autoware framework is something like (the "launch_control" variable enable or disable the inclusion of the control launch file):

<group scoped="false">
    <!-- Vehicle -->
    <let name="launch_vehicle_interface" value="false" if="true"/>
    <let name="launch_vehicle_interface" value="true" unless="true"/>

    <include file="$(find-pkg-share autoware_launch)/launch/autoware.launch.xml">
    <!-- Common -->
      <arg name="map_path" value="$(var map_path)"/>
      <arg name="vehicle_model" value="sample_vehicle"/>
      <arg name="sensor_model" value="sample_sensor_kit"/>
      <arg name="data_path" value="$(var data_path)"/>
      <!-- launch module preset -->
      <arg name="planning_module_preset" value="default"/>

      <!-- Modules to be launched -->
      <arg name="launch_vehicle" value="true"/>
      <arg name="launch_system" value="true"/>
      <arg name="launch_sensing_driver" value="true"/>
      <arg name="launch_map" value="true"/>

      <arg name="launch_sensing" value="false"/>
      <arg name="launch_localization" value="false"/>
      <arg name="launch_perception" value="false"/>

      <arg name="launch_planning" value="true"/>
      <arg name="launch_control" value="false"/>
      <arg name="launch_api" value="true"/>

      <!-- Vehicle -->
      <arg name="launch_vehicle_interface" value="$(var launch_vehicle_interface)"/>
      <!-- System -->
      <arg name="system_run_mode" value="planning_simulation"/>
      <arg name="launch_system_monitor" value="$(var launch_system_monitor)"/>
      <arg name="launch_dummy_diag_publisher" value="$(var launch_dummy_diag_publisher)"/>
      <arg name="diagnostic_graph_aggregator_graph_path" value="$(find-pkg-share autoware_launch)/config/system/diagnostic_graph_aggregator/autoware-psim.yaml"/>
      <!-- Map -->
      <arg name="lanelet2_map_file" value="$(var lanelet2_map_file)"/>
      <arg name="pointcloud_map_file" value="$(var pointcloud_map_file)"/>
      <!-- Tools -->
      <arg name="rviz" value="$(var rviz)"/>
      <arg name="rviz_config" value="$(var rviz_config)"/>
      <arg name="rviz_respawn" value="$(var rviz_respawn)"/>
      <!-- Auto mode setting-->
      <arg name="enable_all_modules_auto_mode" value="$(var enable_all_modules_auto_mode)"/>
      <arg name="is_simulation" value="true"/>
    </include>
  </group>

Step Followed to launch the experiment

I am doing the following:

1. Start Autoware:
   1a. Start Autoware with all the main nodes (without the control node) on the workstation
   1b. Start the control node on the embedded board
2. Place the vehicle on the map
3. Assign a 2D target and wait for the plan to be generated and visible on the map
4. Press the "Auto" button to let Autoware drive the simulated vehicle along the planned path.

When the control node is spawned on the embedded board , I cannot accomplish the step 4.

Debugging Attempt:

• Everything runs perfectly on the workstation when the control node is spawned on the same machine running the Autoware framework (by enabling the inclusion of the tier4_control_components.xml and skipping the step 1b).
• I disabled the firewall on both the machines to be sure that it was not a communication issue;
• I checked that I can see the ros2 nodes and topics coming from the other machine (the workstation sees the ros2 components on the embedded board and vice-versa);
• I am able to see the ros2 nodes and topics flowing on the shared network;

Following some discussions found online, where the "Auto" button was locked as it is in my case, I tried to gather further information running rqt and visualizing the graph debugger: I noticed that there is something wrong with the control node but I am not able to understand where is the problem.

I suspect I am doing something stupid somewhere in the setup...but I am still trying to figure it out. Any help from experienced users will be very appreciated.

I attach the log recorded on the Workstation (ws.log) and the one recorded on the embedded board (embedded.log).
I also attach the full launch files I am using to run the demo: Workstation side (ws.launch.txt) and Embedded board side (ctrl.launch.txt). I also attach a screenshot with the output of the rqt tool.

ctrl.launch.txt
ws.launch.txt

ws.log
embedded.log

Best Regards

Edit: Title

[rt-2pm2]

I tried to debug the problem, by simplifying the experiment.

I realized that the issue is still present even when the partitioning of the autoware components are spawn on the same machine (so launching the two partitioned launchfiles on the same machine).

Looking for the /system/operation_mode/availability topic I see

stop: true
autonomous: false
local: true
remote: true
emergency_stop: true
comfortable_stop: false
pull_over: false

That information is published by the node /system/aggregator which is the implementation of the diagnostic_graph_aggregator

If I got it right, the "unit" reporting the state of the "/autoware/modes/autonomus" is not returning "DiagnosticStatus::OK".
Reading about the diagnostic_graph, I find that

The diagnostic graph aggregator node subscribes to diagnostic array and publishes aggregated diagnostic graph

I tried to log the /diagnostics, /diagnostics_graph/* topics, but I do not see any reference to "autonomous" apart when I get the rejected attempts to switch to autonomy.

Is there anyone who can suggest me how to speed up the tracing of the issue?