Some msgs are dropped #103
Comments
|
Hi there, I am not sure what that plot is showing exactly, since on the x-axis there seems to be a 1e10 scaling? Can you please explain what is shown and if this is live data or from a bag file? Which tool did you use to visualize or record the data? Some ROS 2 tools are known to have problems with high rate data, e.g. here. Also, unreliable connections may have side effects. Also, which interface of the AsteRx is used? Serial baud rates have to be set appropriately for high rate data. |
|
Sorry for my bad explanation.
We though that the problem was the software that we were using to plot the data. We have tried plotjuggler, Floxglove and our own script to plot data (On real time and using rosbags ). Same result in each plotter. We are using ethernet interface to get the data. |
|
Thank you for the clarification. I assume you are running the driver with ROS timestamps, i.e. Counting the data points in your foxglove screenshot, there are 20 withing 0.2 s, which is 100 Hz data rate. So it seems you are not loosing data but the rate ist fluctuating. This could be caused by a highly loaded network. Are there other devices in your network like GigE cameras or LiDARs producing a lot of UDP traffic? Running |
|
Exactly, we are running using ROS timestamps.
|
|
I am sorry but in that case I am a little clueless right now. Some jitter of a few % is normal but I haven't experienced it to that extent. Are you using TCP and could you try with UDP for comparison (parameter |
|
My actual configuration is this one: How can I change stream_device? |
|
You have a lot of publishers active, which means you pose a high demand on the AsteRx and transmission bandwidth. Maybe you can deactivate some publishers if you do not need them. E.g., You can setup the stream device for UDP like this: You should be aware that UDP is meant for low latency transmission of a few topics. From the manual: Note that the UDP implementation is meant to be used with small data volumes and low update rates. It is the user’s responsibility to only enable short messages at low rate when using UDP, in order to prevent throughput degradation of the network. Nevertheless, I have used it with high update rates without any problem but it is advisable to watch the CPU load. |
|
thank you @thomasemter for looking into this. Unfortunately, we're still experiencing issues. Sometimes it works very well, with ROS messages on topic Today we tested with several devices. All the tried configurations reproduced the same undesired result of having a gap of at least 20 ms somewhere in every subsequent 100 messages. We tried two routers, two network switches, two different laptops with a different OS version (Ubuntu 20.04 and 22.04) and ROS distributions (Foxy and Humble), four ways of connecting to the router (native gigabit port, two different USB-to-ethernet adapters and wifi). Internet access was disabled on the router side, the CPU load on the computer was negligible (< 10 %) and only two devices were connected to the router's switch (the Septentrio and our computer running ROS). Capturing the TCP traffic with Wireshark showed that now and then there's a long delay - e.g. 80 ms - between subsequent TCP packets. Enabling UDP mode caused UDP packages to show up in Wireshark, however the Septentrio node didn't publish any messages on the I assume we're making some error in a configuration somewhere, however it is unclear to me where. We would greatly appreciate any suggestions on how to resolve this. In case there is a need for more information from our side, please let know. Further information: The receiver firmware is 1.4.1 and for the ROS node we run the latest version from GitHub compiled from source. We launched the Septentrio node using For using UDP, we added the following below |
|
Thank you for your thorough testing and sorry you are still experiencing this issue. I am still a little clueless what could be the culprit. Just to make sure: By latest version of the driver, do you mean 1.3.1?
|
Probably, the commit log shows v1.3.1, HEAD commit is 207c37b.
Good idea, just tried this (static IPs on both and wifi off). Now UDP does work most of the time and when it works, messages are published every 10 +- 1 ms. However, TCP performs worse than before, there are still large gaps and per second only 90...99 messages are received instead of 100 (see
I'll look whether we have a cable for that.
In TCP mode: So intervals range from 1 to 76 ms. The header timestamps show similar intervals. Here's an excerpt from the TCP package log from Wireshark, with large intervals between packets 496-497 and 521-522 (packets 497 and 522 are also considerably longer): |
|
Thank you for the details. I just tested it with my SBi with TCP via RNDIS and with nearly all publishers activated. With a real ethernet connection max is 0.039 s which is much worse but still not as high as you are experiencing. With UDP both are within 10 +/- 2 ms. Since the latency jitter for TCP is much lower via RNDIS, I do not think the problem lies in the driver or ROS. |
|
I have consulted with Septentrio's support and there is indeed a higher latency when using TCP via real ethernet. It is caused by the implementation in the lower level of the firmware. An improvement is being worked on, which will likely be incorporated in the next AsteRx SBi3 firmware release. For lowest latency operation it is still recommended to go with UDP. |
|
Great that you got in touch with their support, looking forward to the next firmware release! We experimented with UDP several times, sometimes it works, however overall it doesn't perform sufficiently reliably, so we prefer TCP. |
|
A new firmware 1.4.3 has been released, which improves the TCP/IP latency. |
Hello,
I bought asterxsbi3 pro and to read the data I use ros2 node. I saw that the topic twits_ins has a rate 100 Hz but checking some plot I saw some frames are dropped As you can see in the picture attached, some msgs have dissapeared.
twist_ins/linear/x
Any advise?
The text was updated successfully, but these errors were encountered: