aarch64: rr crashes immediately upon replay -e

[BenWibking]

On Fedora 38 aarch64 (Apple Silicon), this can be reproduced with:

[benwibking@fedora ~]$ rr_install/bin/rr record echo "Hello, world"
rr: Saving execution to trace directory `/home/benwibking/.local/share/rr/echo-2'.
Hello, world
[benwibking@fedora ~]$ rr_install/bin/rr replay -e
[ERROR src/Registers.cc:299:maybe_print_reg_mismatch()] x[30] 0x41ffffb0998d40 != 0x7cffffb0998d40 (replaying vs. recorded)
[FATAL src/Registers.cc:409:compare_register_files()] 
 (task 24832 (rec:24825) at time 7)
 -> Assertion `!bail_error || match' failed to hold. Fatal register mismatch (ticks/rec:2748/2748)
Tail of trace dump:
=== Start rr backtrace:
rr_install/bin/rr(_ZN2rr9GdbServer15emergency_debugEPNS_4TaskE+0xe0)[0x48b320]
rr_install/bin/rr[0x49cc1c]
rr_install/bin/rr[0x49d81c]
rr_install/bin/rr(_ZN2rr9Registers22compare_register_filesEPNS_10ReplayTaskEPKcRKS0_S4_S6_NS_16MismatchBehaviorE+0x90)[0x518584]
rr_install/bin/rr(_ZN2rr10ReplayTask13validate_regsEj+0x164)[0x52dc04]
rr_install/bin/rr(_ZN2rr13ReplaySession13enter_syscallEPNS_10ReplayTaskERKNS0_15StepConstraintsE+0x2b4)[0x51f504]
rr_install/bin/rr(_ZN2rr13ReplaySession11replay_stepERKNS0_15StepConstraintsE+0x104)[0x522c04]
rr_install/bin/rr(_ZN2rr14ReplayTimeline19replay_step_forwardENS_10RunCommandE+0x84)[0x53b748]
rr_install/bin/rr(_ZN2rr9GdbServer12serve_replayERKNS0_15ConnectionFlagsE+0x70)[0x486e00]
rr_install/bin/rr[0x5193a4]
rr_install/bin/rr(_ZN2rr13ReplayCommand3runERSt6vectorINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESaIS7_EE+0x444)[0x519d64]
rr_install/bin/rr(main+0x168)[0x433998]
/lib64/libc.so.6(+0x30598)[0xffffb6730598]
/lib64/libc.so.6(__libc_start_main+0x9c)[0xffffb6730670]
rr_install/bin/rr(_start+0x30)[0x434f30]
=== End rr backtrace
Launch gdb with
  gdb '-l' '10000' '-ex' 'set sysroot /' '-ex' 'target extended-remote 127.0.0.1:24832' /home/benwibking/.local/share/rr/echo-2/mmap_clone_5_echo

[Keno]

Is pointer authentication enabled in your kernel? rr currently doesn't capture those keys, I don't think.

[BenWibking]

Is pointer authentication enabled in your kernel? rr currently doesn't capture those keys, I don't think.

I'm using Fedora Asahi, so I'm not 100% sure, but it appears that pointer authentication has been in the upstream Fedora ARM64 kernel since Fedora 33 (https://fedoraproject.org/wiki/Changes/Aarch64_PointerAuthentication).

Have users on Apple Silicon built their own kernels to use rr? Or does only Fedora enable pointer authentication by default?

[Keno]

The apple silicon support was developed with an early version of the Asahi kernel, which I'm fairly sure didn't have PAC enabled. I don't know what Asahi Fedora remix is doing, but I wouldn't be surprised if that got enabled in the process.

[BenWibking]

I'll rebuild the kernel and see it works.

[yuyichao]

I think this is probably not related to early vs later asahi but that arch Linux kernel doesn’t have this enabled vs fedora that does.

[Keno]

I don't think fixing this properly is that bad, we just need to dump the NT_ARM_PACA_KEYS/NT_ARM_PACG_KEYS regsets into the trace.

[Keno]

Try #3569.

[BenWibking]

It works!