[Bug] Pixhawk 6C CAN Interface doesn't recover from any bus error

[igalloway]

Describe the bug

CAN driver for Pixhawk 6C does not recover from CAN bus errors.
Simple things like a slow to power up CAN device or unplugging and replugging a CAN device (GPS ) will cause CAN BUS to stop functioning. Hard reboot required.

To Reproduce

• Start Pixhawk 6C with DroneCAN M8N attached

• DroneCAN M8N works and RGB led goes on

• Disconnect DroneCAN M8N GNSS

• Reconnect DroneCAN M8N GNSS

• RGB led doesn’t, work

• Pixhawk 6C shows errors

•      CAN1 status:
  
       HW errors: 1071
  
       IO errors: 1071
  

Restarting uavcan through cli doesn’t work.
Pixhawk 6C has to be manually restarted to get the CAN GNSS to work.

Expected behavior

Recovery from CAN error. For example unplugging and replugging CAN device, or a device that is slower to power up/respond should be able have CAN bus work. Currently the CAN bus completely fails.

Screenshot / Media

No response

Flight Log

None

Software Version

Main

Flight controller

Pixhawk 6C

Vehicle type

None

How are the different components wired up (including port information)

CAN connected GPS or other device

Additional context

No response

[henrykotze]

What PX4 version are you using? We had similar issues where during boot up, our ESC shorts the CAN H and CAN L lines, causing the uavcan driver to hang. This was resolved in the following PR: #21701

[vincentpoont2]

@davids5 Any idea what can be causing this?

We have a S32K1 CAN GPS and with the Pixhawk 6C, the GPS wouldn't boot up UNLESS we power up the GPS first, then the FC. It doesn't happen on the 6X.

[henrykotze]

@vincentpoont2 This sounds very similar to our experience, where we first had to boot our CAN ESC's before switching on our FMU.

[davids5]

I think it is caused because the driver needs to reset BUSS OFF. Why it happens on 6C and not on 6X is probably the power up time of the 5V supply. @vincentpoont2 if you have HW falls and can ship it to me I can have a look.

[vincentpoont2]

@vincentpoont2 This sounds very similar to our experience, where we first had to boot our CAN ESC's before switching on our FMU.

Thanks! Looks like the RC1/Master does behave differently

Right now S32K1 GPS boot up would take ~1 minute after being power on before RGD LED comes on and PX4 to recognize the CAN node. This delay happens on both 6C & 6X.

@PetervdPerk-NXP would you be able test it RC1 or master and see if you have any idea what is causing the delay?

[davids5]

@vincentpoont2 Where is the source to s32k1-gps @ 73f9e63 ?

HW arch: NXP_UCANS32K146
PX4 git-hash: 73f9e6348d946c5ed41aa50a212892f07c4124eb
PX4 version: 1.14.0 0 (17694720)
PX4 git-branch: s32k1-gps
OS: NuttX
OS version: Release 11.0.0 (184549631)
OS git-hash: 3f77354c0dc88793a47ff3b57595195ab45f7ba9
Build datetime: Aug  7 2023 06:10:19
Build uri: localhost
Build variant: default
Toolchain: GNU GCC, 9.3.1 20200408 (release)
PX4GUID: 0007001dffffffffffff4e53375040170019
MCU: S32K146, rev. 0

[davids5]

@vincentpoont2 This sounds very similar to our experience, where we first had to boot our CAN ESC's before switching on our FMU.

Thanks! Looks like the RC1/Master does behave differently

Right now S32K1 GPS boot up would take ~1 minute after being power on before RGD LED comes on and PX4 to recognize the CAN node. This delay happens on both 6C & 6X.

@PetervdPerk-NXP would you be able test it RC1 or master and see if you have any idea what is causing the delay?

The S32K1 GPS is not acking the CAN messages (it is stuck in the bootloader and not booting) - if you add a CAN Adaptor that will ACK the message it will boot in 5 Sec or so. So the code needs some love

[davids5]

@vincentpoont2 @igalloway @PetervdPerk-NXP
Please test #22247

[davids5]

I merged the PR because @AlexKlimaj Verified his use case.

@vincentpoont2 - Please test main and reopen this issue if you find any issue with it.