Linking problem with dynamic libs on iOS

[aeddi]

We have a problem linking the different dynamic libs (libssl, libcrypto, libevent, zlib) required by the go-libtor package with our iOS app (arm64/iphoneos and amd64/iphonesimulator).

We lack time to investigate more and as we say in French: "Better one who knows than ten who are trying to find a solution".

To help find the problem, we have created a separate repo to do sanity checks, there are several branches:

1. This one contains a simple iOS app calling a go package compiled with gomobile calling itself functions from different libs (openssl, libevent, etc...).
2. This one is similar but uses a React-Native app.
3. The last one contains a React-Native app calling the libtor package compiled with gomobile.

The dynamic libs are the same on every branches, the first two work properly and the last one doesn't work (build ok, linking ok but crash during call to an openssl function). This suggests that the problem is probably due to an incompatibility between the dynamic libs and the go-libtor package.

@Jorropo will maybe continue to investigate on this issue, in any case, he can detail the steps he used to compile the dynamic libs. On my side I took care of:

• merging the arm64 and amd64 versions using lipo. e.g:

for lib in arm64/*.dylib; do
    lipo "arm64/${lib##*/}.dylib" "amd64/${lib##*/}.dylib" -create -output "fat/${lib##*/}.dylib"
done

• setting the correct rpath using install_name_tool. e.g:

for lib in *.dylib; do
    install_name_tool -id "@rpath/$lib" $lib
done

install_name_tool -change /tmp/torpref/lib/libcrypto.1.1.dylib "@rpath/libcrypto.1.1.dylib" libssl.1.1.dylib`
etc...

Note: these steps are not problematic as it works well on both targeted architectures and the linker is able to load the dylibs during runtime.

The following error occurs when pressing the Test button which trigger a call to a go-libtor test function (sorted from the bottom to the top of the stack, see left panel):

Steps to reproduce the problem on third branch:

• run make RNApp/node_modules RNApp/ios/Pods framework/Cgo.framework
• open RNApp/ios/RNApp.xcworkspace with XCode
• run app in XCode

[aimxhaisse]

Spent some time digging:

React statically links with a different libssl version than libtor:

$ strings ./RNApp/ios/Pods/OpenSSL-Universal/ios/lib/libssl.a | grep 'OpenSSL 1.
OpenSSL 1.0.2u 20 Dec 2019

$ strings ./openssl/lib/libssl.dylib | grep 'OpenSSL 1.'
OpenSSL 1.1.1i-dev xx XXX xxxx

The crash is while accessing libssl.1.1.dylib`TLSv1_2_enc_data @ 0x11199f3e0.

Looking at the asm around the crash, we are in function SSL_CTX_new:

RNApp`SSL_CTX_new:
    0x10230ab60 <+0>:    pushq  %rbp
    0x10230ab61 <+1>:    movq   %rsp, %rbp
    0x10230ab64 <+4>:    pushq  %r14
    0x10230ab66 <+6>:    pushq  %rbx
    0x10230ab67 <+7>:    testq  %rdi, %rdi
    0x10230ab6a <+10>:   je     0x10230af89               ; <+1065>
    0x10230ab70 <+16>:   movq   %rdi, %r14
    0x10230ab73 <+19>:   callq  0x10230ccf0               ; SSL_get_ex_data_X509_STORE_CTX_idx
    0x10230ab78 <+24>:   testl  %eax, %eax
    0x10230ab7a <+26>:   js     0x10230afac               ; <+1100>
    0x10230ab80 <+32>:   leaq   0x5fb446(%rip), %rsi      ; "ssl_lib.c"
    0x10230ab87 <+39>:   movl   $0x320, %edi              ; imm = 0x320 
    0x10230ab8c <+44>:   movl   $0x778, %edx              ; imm = 0x778 
    0x10230ab91 <+49>:   callq  0x1021d25c0               ; CRYPTO_malloc
    0x10230ab96 <+54>:   testq  %rax, %rax
    0x10230ab99 <+57>:   je     0x10230afcd               ; <+1133>
    0x10230ab9f <+63>:   movq   %rax, %rbx
    0x10230aba2 <+66>:   movl   $0x320, %edx              ; imm = 0x320 
    0x10230aba7 <+71>:   movq   %rax, %rdi
    0x10230abaa <+74>:   xorl   %esi, %esi
    0x10230abac <+76>:   callq  0x102865896               ; symbol stub for: memset
    0x10230abb1 <+81>:   movq   %r14, (%rbx)
    0x10230abb4 <+84>:   movq   $0x0, 0x18(%rbx)
    0x10230abbc <+92>:   movl   $0x2, 0x40(%rbx)
    0x10230abc3 <+99>:   movq   $0x5000, 0x28(%rbx)       ; imm = 0x5000 
    0x10230abcb <+107>:  movq   $0x0, 0x30(%rbx)
    0x10230abd3 <+115>:  movq   $0x0, 0x38(%rbx)
    0x10230abdb <+123>:  callq  *0xc0(%r14)

[...]

    0x10230ad88 <+552>:  je     0x10230aff0               ; <+1168>
    0x10230ad8e <+558>:  leaq   0x5fb296(%rip), %rdi      ; "ssl2-md5"
    0x10230ad95 <+565>:  callq  0x10226b210               ; EVP_get_digestbyname
    0x10230ad9a <+570>:  movq   %rax, 0xe0(%rbx)
    0x10230ada1 <+577>:  testq  %rax, %rax
    0x10230ada4 <+580>:  je     0x10230b041               ; <+1249>
    0x10230adaa <+586>:  leaq   0x5fb283(%rip), %rdi      ; "ssl3-md5"
    0x10230adb1 <+593>:  callq  0x10226b210               ; EVP_get_digestbyname
    0x10230adb6 <+598>:  movq   %rax, 0xe8(%rbx)
    0x10230adbd <+605>:  testq  %rax, %rax
    0x10230adc0 <+608>:  je     0x10230b05f               ; <+1279>
    0x10230adc6 <+614>:  leaq   0x5fb270(%rip), %rdi      ; "ssl3-sha1"
    0x10230adcd <+621>:  callq  0x10226b210               ; EVP_get_digestbyname

Checking the code of openssl between versions 1.1.1 and 1.0.2, this asm corresponds to the 1.0.2 implementation (the lib checks for "ssl2-md5", "ssl3-md5", "ssl3-sha1" in 1.0.2, whereas in 1.1.1 it checks for "ssl3-md5" and "ssl3-sha1"):

So, libtor's code which uses headers for OpenSSL 1.1.1 & calls some symbols from 1.0.2.

Now, if we look at the symbol for TLSv1_2_enc_data, we see the opposite, 1.1.1 is the one used:

(lldb) x/64 0x1052533e0 # (address of TLSv1_2_enc_data)
0x1052533e0: 0x051d8c10 0x00000001 0x051da6b0 0x00000001
0x1052533f0: 0x05235370 0x00000001 0x05235c00 0x00000001
0x105253400: 0x052347e0 0x00000001 0x05235760 0x00000001
0x105253410: 0x05243242 0x00000001 0x0000000f 0x00000000 # TLS_MD_CLIENT_FINISH_CONST,  TLS_MD_CLIENT_FINISH_CONST_SIZE
0x105253420: 0x05243252 0x00000001 0x0000000f 0x00000000 # TLS_MD_SERVER_FINISH_CONST,  TLS_MD_SERVER_FINISH_CONST_SIZE
0x105253430: 0x052362b0 0x00000001 0x05235ee0 0x00000001
0x105253440: 0x00000017 0x00000000 0x051dfee0 0x00000001

The TLS_MD_... entries start after 6 adresses in 1.1.1 whereas in 1.0.2 it starts after 6 adresses followed by an int and another address:

So we have 1.0.2 code call to data layed out for 1.1.1.

So there is some weird interleaving of symbols going both ways.

I've spent some time digging in link options to see if there is a way to prevent this but didn't find a solution there.

One sad alternative would be to have libtor use the exact same version as the react one, not sure if this is acceptable.

By statically linking openssl in libtor we get a different crash

I didn't dig into it but maybe it is an easier path, just pasting here for posterity:

--- a/Makefile
+++ b/Makefile
@@ -22,7 +22,9 @@ framework/Cgo.framework: $(wildcard cgo/*)
        go run golang.org/x/mobile/cmd/gomobile init
        GO111MODULE=on $(library_path) $(cpath) \
                go run golang.org/x/mobile/cmd/gomobile bind \
+                       -x -v \
                        -o $@ \
+                       -tags "staticOpenssl staticZlib staticLibevent" \
                        -target ios \
                        ./cgo

(lldb) bt
* thread #3, stop reason = EXC_BAD_ACCESS (code=1, address=0x408c000)
    frame #0: 0x000000000408c000
  * frame #1: 0x000000010cf0a85d RNApp`RSA_generate_multi_prime_key + 77
    frame #2: 0x000000010cf0a7f8 RNApp`RSA_generate_key_ex + 104
    frame #3: 0x000000010d1a0398 RNApp`crypto_pk_generate_key_with_bits + 232
    frame #4: 0x000000010d15905f RNApp`init_keys_client + 79
    frame #5: 0x000000010d1591ef RNApp`init_keys + 95
    frame #6: 0x000000010cf466df RNApp`run_tor_main_loop + 47
    frame #7: 0x000000010cf46cb5 RNApp`tor_run_main + 597
    frame #8: 0x000000010d203d39 RNApp`_cgo_bcc88fe7c62b_Cfunc_tor_run_main + 41
    frame #9: 0x000000010cc776f0 RNApp`runtime.asmcgocall at asm_amd64.s:655
    frame #10: 0x000000010ce72fd9 RNApp`berty.tech/go-libtor/libtor._Cfunc_tor_run_main at _cgo_gotypes.go:183
    frame #11: 0x000000010ce73e0f RNApp`berty.tech/go-libtor/libtor.(*embeddedProcess).Start.func5.3 at libtor.go:92
    frame #12: 0x000000010ce73f08 RNApp`berty.tech/go-libtor/libtor.(*embeddedProcess).Start.func5 at libtor.go:92
    frame #13: 0x000000010cc77f51 RNApp`runtime.goexit at asm_amd64.s:1357
    frame #14: 0x000000010ce73e0f RNApp`berty.tech/go-libtor/libtor.(*embeddedProcess).Start.func5.3 at libtor.go:92
    frame #15: 0x000000010ce73f08 RNApp`berty.tech/go-libtor/libtor.(*embeddedProcess).Start.func5 at libtor.go:92
    frame #16: 0x000000010cc77f51 RNApp`runtime.goexit at asm_amd64.s:1357
    frame #17: 0x000000010ce73f08 RNApp`berty.tech/go-libtor/libtor.(*embeddedProcess).Start.func5 at libtor.go:92
    frame #18: 0x000000010cc77f51 RNApp`runtime.goexit at asm_amd64.s:1357
    frame #19: 0x000000010cc77f51 RNApp`runtime.goexit at asm_amd64.s:1357

It crashes in an unmapped memory location:

(lldb) memory region 0x408c000
[0x000000000408c000-0x0000000104527000) ---
(lldb) 

Could this one be a different sort of crash?

[aeddi]

Thank you very much for digging @aimxhaisse 👍
Ok that confirms what I thought, thanks to the details you gave me, I have to confirm if the app of the react-native-app branch doesn't crash because it doesn't call the same symbol or if the problem comes from the go-libtor build which for some reason calls symbols from 1.0.2 while it is linked with a dynamic lib in version 1.1.1.
I'll test it ASAP.

[aeddi]

Ok so I tested calling SSL_CTX_new(TLS_method()) from the react-native-app branch (without go-libtor) and it crashes too.
So I'm going to try what suggests @aimxhaisse and @steeve (on our Discord) and manage to get only one version of Openssl for react-native and go-libtor.

[aeddi]

It works when using the same version for both React-Native and go-libtor.
Thanks for your help @aimxhaisse and @steeve :)