Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cannot Enable eRob [ZeroErr CIA402 Motor] DC Mode and Random OP State Entry with ethercat_driver_ros2 #154

Open
ZeroErrControl opened this issue Dec 26, 2024 · 5 comments

Comments

@ZeroErrControl
Copy link

ZeroErrControl commented Dec 26, 2024

Description

Environment Details

  • OS: Ubuntu 20.04
  • ROS 2 Version: Humble
  • Motor: ZeroErr eRob (CIA402 protocol compliant)
  • Driver: ethercat_driver_ros2
  • Master: IGH EtherCAT Master (for comparison)

I am using ethercat_driver_ros2 to integrate ZeroErr's eRob servo motor (compliant with the CIA402 protocol) with MoveIt for motion control. However, the following issues were encountered:

  1. eRob cannot reliably enter Distributed Clock (DC) mode.
  2. The slave shows random behavior when entering OP state, often remaining in Safe OP + Error instead of OP state.

These issues prevent the seamless use of ethercat_driver_ros2 to control the eRob motor.


Experiments and Observations

Experiment 1: Removing DC Configuration Code

I removed the following code block from ec_master.cpp:

if (slave->assign_activate_dc_sync()) {
    struct timespec t;
    clock_gettime(CLOCK_MONOTONIC, &t);
    ecrt_master_application_time(master_, EC_NEWTIMEVAL2NANO(t));
    ecrt_slave_config_dc(
      slave_info.config,
      slave->assign_activate_dc_sync(),
      interval_,
      interval_ - (t.tv_nsec % (interval_)),
      0,
      0);
}

Results:

eRob successfully entered OP state.
Running ethercat upload -p0 0x1c32 1 returned:

0x0001 1

This indicates that eRob remained in Sync Manager (SM) mode rather than DC mode.
Conclusion:
Removing the DC configuration allows eRob to enter OP state but prevents DC mode activation.

Experiment 2: Forcing DC Configuration

I removed the condition ethercat_driver_ros2/ethercat_interface/src/ec_master.cpp check and forced the DC configuration code to execute:

struct timespec t;
clock_gettime(CLOCK_MONOTONIC, &t);
ecrt_master_application_time(master_, EC_NEWTIMEVAL2NANO(t));
ecrt_slave_config_dc(
  slave_info.config,
  slave->assign_activate_dc_sync(),
  interval_,
  interval_ - (t.tv_nsec % (interval_)),
  0,
  0);

Results:

eRob successfully entered DC mode.
Running ethercat upload -p0 0x1c32 1 returned: 0x0002 2

This confirms that eRob was in DC mode.
However, eRob failed to enter OP state and remained in Safe OP + Error state.
Kernel logs showed the following messages:

[76477.776110] EtherCAT 0: Starting EtherCAT-OP thread.
[76478.371933] EtherCAT WARNING 0: 3 datagrams TIMED OUT!
[76478.371944] EtherCAT WARNING 0: 4 datagrams UNMATCHED!
[76484.748536] EtherCAT WARNING 0-0: Slave did not sync after 5000 ms.
[76494.198581] EtherCAT ERROR 0-0: Failed to set OP state, slave refused state change (SAFEOP + ERROR).
[76494.208694] EtherCAT ERROR 0-0: AL status message 0x001A: "Synchronization error".

Conclusion:

Forcing DC configuration enables DC mode but causes synchronization errors, preventing OP state entry.

Another observation: When forced into DC mode, the six-axis system can successfully enter DC mode and transition to OP state, but the single-axis and two-axis systems cannot enter the OP state.

Experiment 3: Using IGH EtherCAT Master

I used a custom IGH EtherCAT master implementation (code link )to control eRob.

Results:

Successfully enabled DC mode.
eRob entered OP state without errors.
All functionalities worked as expected.

Conclusion:

Using the IGH EtherCAT master confirms that the issue is specific to ethercat_driver_ros2.

Analysis

DC Configuration Implementation:

  • The interval_ parameter and the shift_time calculation in ecrt_slave_config_dc might be incorrect, leading to timing mismatches.

  • The master application time (ecrt_master_application_time) may not align with the slave's reference clock, causing synchronization issues.

  • The DC configuration might fail to fully activate all required SYNC managers (e.g., SYNC0).
    Synchronization Errors:

  • Kernel logs show AL status message 0x001A ("Synchronization error"), indicating that the master and slave clocks are not properly synchronized.
    Contrast with IGH EtherCAT Master:

  • The same hardware and configuration worked perfectly with the IGH EtherCAT master, suggesting that ethercat_driver_ros2 has issues in handling DC mode and synchronization.

Recommendations

Verify and Fix DC Configuration:

  • Ensure interval_ and shift_time parameters are correctly calculated based on the desired cycle time and reference clock.
    Check that ecrt_master_application_time properly aligns with the slave's reference clock.
    Confirm that both SYNC0 and SYNC1 are properly configured.
    Improve Synchronization Logic:

  • Add debug logs to verify the execution and effects of ecrt_master_sync_reference_clock and ecrt_master_sync_slave_clocks.
    Validate that the master application time is set before attempting synchronization.
    Enhance Compatibility:

  • Review and ensure ethercat_driver_ros2 fully supports CIA402-compliant devices like eRob.
    Add configuration options or examples specifically tailored for CIA402 motors.

@ZeroErrControl ZeroErrControl changed the title Cannot Enable eRob [ZeroERR CIA402 Motor] DC Mode and Random OP State Entry with ethercat_driver_ros2 Cannot Enable eRob [ZeroErr CIA402 Motor] DC Mode and Random OP State Entry with ethercat_driver_ros2 Dec 26, 2024
@ZeroErrControl
Copy link
Author

I have preliminarily resolved the issue where the ethercat_ros2 interface fails to bring eRob into the OP (Operational) state. The detailed solution is provided in the following project: Project Link.

@pzanne
Copy link
Member

pzanne commented Jan 6, 2025

Hello ZeroErrControl,
Thank you for your feedback,

Improving the behaviour of the distributed clock is in our development plan, and we could use your use case to make progress on this point. The cia402 drivers that we've been able to test (technosoft, elmo, synapticon, maxon) so far haven't any problems for switching to operational mode in DC mode. Would it be possible to access your servomotor for testing purposes (remotely, for example)?

Your kernel logs in experiment 2 show ethercat frame losses (DATAGRAM timed out and DATAGRAM unmatched). Could you test the branch humble-issue-147 and tell me if this changes anything when switching to operational mode?

Could you also send me your kernel logs for experiment 3?
Best
Philippe

@ZeroErrControl
Copy link
Author

Dear Philippe,

Thank you very much for your detailed response and for seriously considering my suggestions.

Based on your feedback, I have formulated the following experimental plan:

Comparison of DC Performance:
We will conduct a comparison experiment between the CIA402 drivers you mentioned (Technosoft, Elmo, Synapticon, Maxon) and our eRob (ZeroErr CIA402 Motor) under Distributed Clock (DC) mode.
Additionally, the idea of remote access to my servo motor is certainly feasible. Could you provide guidance on how to proceed with the connection and continuous operation?

Testing the Humble-issue-147 Branch:
I will test the branch you mentioned and carefully observe and record any changes during the switch to operational mode.

Kernel Logs for Experiment 3:
I will repeat Experiment 3 and organize the kernel logs to share them with you for further analysis.

These experiments will require some time to complete, but I will ensure the results are well-organized and sent to you as soon as possible.

Thank you again for your support, and I look forward to continuing this collaboration!

Best regards,
ZeroErrControl

@pzanne
Copy link
Member

pzanne commented Jan 20, 2025

Hi @ZeroErrControl

We've carried out some tests, and it's possible that we've found the source of the clock synchronisation problem.
It is possible that the problem is caused by the cycle time management in ros2_control. If a cycle time is too long in relation to the period, the read/update/write events are buffered in ros2_control. As soon as ros2_control takes over, it tries to empty this buffer, leading to temporal instability when calling the read/write functions of our ethercat driver (very bad for the DC synchronization).
We are currently looking for a solution to this problem. As soon as we have something working we'll get back to you.
Best
Philippe

@ZeroErrControl
Copy link
Author

Dear Philippe,

Your identified issue indeed matches the problem we've observed. When the DC mode fails to initialize, the slave falls back to SM mode, and it seems the slave's configuration gets reset. Thank you very much for pinpointing the issue, and we look forward to your solution.

Best regards,
ZeroErrControl

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants