Portenta X8 RPC tutorial is not working at all

[michelgokan]

I followed this tutorial to establish a "Data Exchange Between Python on Linux & Arduino Sketch": https://docs.arduino.cc/tutorials/portenta-x8/python-arduino-data-exchange/

I've tried going through the tutorial with both the latest firmware image (861 (Release arduino-91.2)) and also 746 version, on my Portenta X8, tried both with and without the mid-carrier board.

I have py-serialrpc running as follows:

bash-5.1$ sudo docker compose up
[+] Building 0.0s (0/0)
[+] Running 2/2
 ✔ Network py-serialrpc_default           Created0.2s
 ✔ Container py-serialrpc-py-serialrpc-1  Created0.1s
Attaching to py-serialrpc-py-serialrpc-1
py-serialrpc-py-serialrpc-1  | server: m4-proxy, port: 5000

this is the output of my sudo journalctl -u m4-proxy -f:

bash-5.1$ sudo journalctl -u m4-proxy -f
Password:
Sorry, try again.
Password:
Jul 30 15:30:23 portenta-x8-200fc209dab6fad9 systemd[1]: Started M4 builtin led forwarder.
Jul 30 15:30:29 portenta-x8-200fc209dab6fad9 m4_proxy[750]: Registering service on port  5005  with functions  map[count:5005 len:5005 tty:5005]
Jul 30 15:31:05 portenta-x8-200fc209dab6fad9 m4_proxy[750]: Registering service on port  5005  with functions  map[count:5005 len:5005 tty:5005]

Here is the Arduino Sketch code according to the python-sensor-rpc mentioned in the tutorial without any sensors:

#include <RPC.h>
#include <SerialRPC.h>
#include <Wire.h>

void setup()
{
    Serial.begin(115200);
    Serial.println("BME680 test on M4");
    Wire.begin();

    Serial.println("Trying to find sensor...");
    
    //RPC.bind("status", []{ return bme.sensorID() == 0x60; });
    RPC.bind("temperature", []{ return 100; });
    RPC.bind("humidity", []{ return 200; });
    RPC.bind("pressure", []{ return 300; });
    RPC.bind("gas", []{ return 400; });
    RPC.bind("altitude", []{ return 500; });

    Serial.println("Starting");
}

void loop()
{
  Serial.print("Temperature = ");
  Serial.print(100);
  Serial.println(" *C");

  Serial.print("Pressure = ");
  Serial.print(200);
  Serial.println(" hPa");

  Serial.print("Humidity = ");
  Serial.print(300);
  Serial.println(" %");

  Serial.print("Gas = ");
  Serial.print(400);
  Serial.println(" KOhms");

  Serial.print("Approx. Altitude = ");
  Serial.print(500);
  Serial.println(" m");

  Serial.println();

  delay(1000);
}

...and here is the python code:

import time
from msgpackrpc import Address as RpcAddress, Client as RpcClient, error as RpcError


# Fixed configuration parameters
port = 8884
publish_interval = 5

# The M4 Proxy address needs to be mapped via Docker's extra hosts
m4_proxy_address = 'm4-proxy'
m4_proxy_port = 5001

def get_data_from_m4():
    """Get data from the M4 via RPC (MessagePack-RPC)

    The Arduino sketch on the M4 must implement the following methods
    returning the suitable values from the attached sensor:

    * `temperature`
    * `humidity`
    * `pressure`
    * `gas`
    * `altitude`

    """

    rpc_address = RpcAddress(m4_proxy_address, m4_proxy_port)

    data = ()

    try:
        rpc_client = RpcClient(rpc_address)
        temperature = rpc_client.call('temperature')

        rpc_client = RpcClient(rpc_address)
        humidity = rpc_client.call('humidity')

        rpc_client = RpcClient(rpc_address)
        pressure = rpc_client.call('pressure')

        rpc_client = RpcClient(rpc_address)
        gas = rpc_client.call('gas')

        rpc_client = RpcClient(rpc_address)
        altitude = rpc_client.call('altitude')

        data = temperature, humidity, pressure, gas, altitude

    except RpcError.TimeoutError:
        print("Unable to retrive data from the M4.")

    return data


if __name__ == '__main__':

    print()
    print("============================================")
    print("==       Portenta X8 Sensor reading       ==")
    print("============================================")
    print()

    try:
        while True:
            data = get_data_from_m4()
            if len(data) > 0:
                print("Temperature: ", data[0])
                print("Humidity: ", data[1])
                print("Pressure: ", data[2])
                print("Gas: ", data[3])
                print("Altitude: ", data[4])
            time.sleep(publish_interval)
    except KeyboardInterrupt:
        print('Stopped.')

Here is the Dockerfile:

FROM python:3.9-alpine AS Build

RUN set -ex \
    && apk add --no-cache --virtual .build-dep \
        git build-base python3-dev

RUN set -ex \
    && git clone https://github.com/kpn-iot/senml-python-library \
    && cd senml-python-library \
    && python3 setup.py bdist_wheel \
    && cd ..

RUN set -ex \
    && apk del .build-dep

FROM python:3.9-alpine

COPY --from=Build /senml-python-library/dist/kpn_senml-1.1.1-py3-none-any.whl ./
COPY requirements.txt ./

RUN set -ex \
    && pip3 --disable-pip-version-check --no-cache-dir install \
        -r requirements.txt \
    && rm ./kpn_senml-1.1.1-py3-none-any.whl \
    && rm ./requirements.txt

WORKDIR /app

COPY src/m4_to_python.py .

ENTRYPOINT [ "python3", "m4_to_python.py"]

...and here is the docker-compose.yml:

services:
  python-sensor-rpc:
    #image: hub.foundries.io/arduino/python-sensor-rpc:latest
    image: python-sensor-rpc:latest
    restart: unless-stopped
    environment:
      - PYTHONUNBUFFERED=1
    volumes:
      - /var/run/secrets:/app/config/
    extra_hosts:
      - "m4-proxy:host-gateway"

I basically haven't changed anything in the original python-sensor-rpc.zip and py-serialrpc.zip files, except for in py-serialrpc I also pip installed backports.ssl_match_hostname and backports.weakref packages. So no change whatsoever in the code.

Here is the error after bringing up the container:

sudo docker compose up
[+] Running 2/2
 ⠿ Network python-sensor-rpc_default                Created 0.2s
 ⠿ Container python-sensor-rpc-python-sensor-rpc-1  Created 0.1ss
Attaching to python-sensor-rpc-python-sensor-rpc-1
python-sensor-rpc-python-sensor-rpc-1  |
python-sensor-rpc-python-sensor-rpc-1  | ============================================
python-sensor-rpc-python-sensor-rpc-1  | ==       Portenta X8 Sensor reading       ==
python-sensor-rpc-python-sensor-rpc-1  | ============================================
python-sensor-rpc-python-sensor-rpc-1  |
python-sensor-rpc-python-sensor-rpc-1  | Unable to retrive data from the M4.
python-sensor-rpc-python-sensor-rpc-1  | Unable to retrive data from the M4.
python-sensor-rpc-python-sensor-rpc-1  | Unable to retrive data from the M4.
python-sensor-rpc-python-sensor-rpc-1  | Unable to retrive data from the M4.
python-sensor-rpc-python-sensor-rpc-1  | Unable to retrive data from the M4.
python-sensor-rpc-python-sensor-rpc-1  | Unable to retrive data from the M4.
python-sensor-rpc-python-sensor-rpc-1  | Unable to retrive data from the M4.
python-sensor-rpc-python-sensor-rpc-1  | Unable to retrive data from the M4.
python-sensor-rpc-python-sensor-rpc-1  | Unable to retrive data from the M4.
python-sensor-rpc-python-sensor-rpc-1  | Unable to retrive data from the M4.
python-sensor-rpc-python-sensor-rpc-1  | Unable to retrive data from the M4.

...I'm now getting this in m4-proxy output:

Jul 30 21:00:13 portenta-x8-200fc209dab6fad9 m4_proxy[827]: resolving  register
Jul 30 21:00:13 portenta-x8-200fc209dab6fad9 m4_proxy[827]: Registering service on port  5005  with functions  map[tty:5005]

Jul 30 21:53:04 portenta-x8-200fc209dab6fad9 m4_proxy[20457]: Serving 172.23.0.2:37126
Jul 30 21:53:04 portenta-x8-200fc209dab6fad9 m4_proxy[20457]: [0 0 [116 101 109 112 101 114 97 116 117 114 101] []]
Jul 30 21:53:04 portenta-x8-200fc209dab6fad9 m4_proxy[20457]: {[148 0 0 171 116 101 109 112 101 114 97 116 117 114 101 144] 0 0}
Jul 30 21:53:04 portenta-x8-200fc209dab6fad9 m4_proxy[20457]: wait for response
Jul 30 21:53:05 portenta-x8-200fc209dab6fad9 m4_proxy[20457]: done
Jul 30 21:53:20 portenta-x8-200fc209dab6fad9 m4_proxy[20457]: Serving 172.23.0.2:37128
Jul 30 21:53:20 portenta-x8-200fc209dab6fad9 m4_proxy[20457]: [0 0 [116 101 109 112 101 114 97 116 117 114 101] []]
Jul 30 21:53:20 portenta-x8-200fc209dab6fad9 m4_proxy[20457]: {[148 0 0 171 116 101 109 112 101 114 97 116 117 114 101 144] 0 0}
Jul 30 21:53:20 portenta-x8-200fc209dab6fad9 m4_proxy[20457]: wait for response
Jul 30 21:53:21 portenta-x8-200fc209dab6fad9 m4_proxy[20457]: done
Jul 30 21:53:36 portenta-x8-200fc209dab6fad9 m4_proxy[20457]: Serving 172.23.0.2:37130
...

I don't see any output in py-serialrpc output (I had to pip install backports.ssl_match_hostname and backports.weakref to make dockerfile compile though):

sudo docker compose up
[+] Running 2/2
 ⠿ Network py-serialrpc_default           Created 0.1s
 ⠿ Container py-serialrpc-py-serialrpc-1  Created 0.1ss
Attaching to py-serialrpc-py-serialrpc-1
py-serialrpc-py-serialrpc-1  | server: m4-proxy, port: 5000

Any ideas what's wrong and why the code is unable to call the RPC functions? I can't even see any of the Serial outputs neither on py-serialrpc output nor Arduino Serial Monitor. Without serial support, this board will be useless for many industrial use cases, including ours.

It's very likely that the m4-proxy is not capturing the RPC.bind from the Arduino Sketch side, because I tried a simple client/server on Python side and it was working okay! I'm not sure what's the exact problem :(