feat: use status backend server (#21550)

* feat: use status backend server

* chore: add env variable STATUS_BACKEND_SERVER_IMAGE_SERVER_URI_PREFIX

* update doc

* fix_: image_server lint issue

doc/use-status-backend-server.md

@@ -0,0 +1,53 @@
+## Solution to use Status Backend Server
+`StatusBackendClient` is the entry point to use Status Backend Server. We need always to call `status-im.setup.status-backend-client/init` whether `STATUS_BACKEND_SERVER_ENABLED` is `1` or not. If it's not enabled, the invocation to functions in `native-module.core` will be delegated to built-in status-go library, otherwise it will be delegated to status-go running in status-backend server. Currently, all functions has usages in `native-module.core` should be supported delegated to. 
+NOTE: not all the native functions used `StatusBackendClient`, only the corresponding functions in `native-module.core` has usages should be supported delegated to ATM.
+
+related [PR](https://github.com/status-im/status-mobile/pull/21550)
+
+## Usage
+### Add environment variables to your local machine:
+```shell
+# enable using status backend server or not, otherwise it will use built-in status-go library
+export STATUS_BACKEND_SERVER_ENABLED=1
+
+#The host should contain an IP address and a port separated by a colon. 
+#The port comes from your running status backend server. 
+#If you run it by PORT=60000 make run-status-backend , then host will likely be 127.0.0.1:60000
+export STATUS_BACKEND_SERVER_HOST="127.0.0.1:60000" 
+
+export STATUS_BACKEND_SERVER_ROOT_DATA_DIR="/path/to/your/root/data/dir" 
+```
+You need to change `STATUS_BACKEND_SERVER_ROOT_DATA_DIR` to your preferred directory and ensure it exists, it should be in absolute path.
+All the db files and log files(requests.log/geth.log) and keystore files etc will be stored in this directory.
+
+### Start the status backend server:
+```shell
+PORT=60000 make run-status-backend
+```
+MAKE SURE the status-backend is checked out to a revision that's at least compatible with the revision in status-mobile/status-go-version.json before starting the server.
+
+For the Android simulator, you need to reverse the port:
+```shell
+adb reverse tcp:60000 tcp:60000
+```
+However, there is restriction when use adb reverse, we use random port for media server. So I'd suggest use "10.0.2.2:60000" as `STATUS_BACKEND_SERVER_HOST`.
+
+### Debug status-go using IDEA
+Assume you've already set up the development environment for status-go, open status-go project use IDEA, run `make generate` with terminal in the status-go project root directory to ensure all the generated files are up to date, then open `cmd/status-backend/main.go` file, navigate to function `main()`, click the green play button on the left of `main()`, choose `Modify Run Configuration`, in `Program arguments` section, add `--address=localhost:60000 --media-https=false`, then click `OK`, finally click the green play button on the left of `main()` again and choose `Debug ...`.
+Basically, you don't have to run `make generate` again and again, just run it once at the beginning. So you can re-run and debug it faster!
+
+## Known Android simulator issues
+- Issue#1: Android simulator may not display images due to TLS certificate validation issues with the image server
+  - solution: use http instead of https for media server with command: `MEDIA_HTTPS=false PORT=60000 make run-status-backend`, currently status-go does not support http for media server. You have to use this draft [PR](https://github.com/status-im/status-go/pull/6060), you also need to set env variable `STATUS_BACKEND_SERVER_IMAGE_SERVER_URI_PREFIX` to "http://10.0.2.2:" so that `image_server.cljs` can work, and to make `/accountInitials` work, you need to copy `Inter-Medium.ttf` to your host machine from the android simulator, let's say you store it in `/Users/mac/Downloads/Inter-Medium.ttf`, then you need to update `get-font-file-ready` manually in `image_server.cljs` to return the correct path so that status backend server can access it.
+- Issue#2: exportUnencryptedDatabaseV2/import-multiaccount does not work for android, probably cause of tech debt, I found it during creating the draft PR.
+- Issue#3: unable to invoke `multiaccounts_storeIdentityImage` to change avatar image.
+  - The reason is that we path the absolute path of the image to the backend server, but the image file is stored in the android simulator. the backend server cannot access it as it runs in the host machine.
+
+If you're using ios simulator, you can skip above issues!
+
+## Details for issue#1 if you're interested
+- we use `react-native-fast-image` which use okhttpclient behind
+- we were using custom cert for https
+- we fetch the custom cert through endpoint `ImageServerTLSCert`
+- we fetched it through built-in status-go before, now we need to fetch it through status backend server
+- we expect `OkHttpClientProvider.setOkHttpClientFactory(StatusOkHttpClientFactory())` to be invoked early(will trigger fetching wrong custom cert via built-in status-go since StatusBackendClient is not initialised yet!) in `MainApplication.kt` before executing clojure code, otherwise we will get black screen after `make run-android` . After deep research, I found there's no way to update the cert okhttpclient used to the correct one return from status backend server

ios/StatusIm/AppDelegate.mm

@@ -29,6 +29,8 @@
 #import <Security/Security.h>
 #import <react/config/ReactNativeConfig.h>
 
+#import "StatusBackendClient.h"
+
 //TODO: properly import the framework
 extern "C" NSString* StatusgoImageServerTLSCert();
 
@@ -188,7 +190,12 @@ + (void)URLSession:(NSURLSession *)session task:(NSURLSessionTask *)task didRece
   NSURLSessionAuthChallengeDisposition disposition = NSURLSessionAuthChallengeCancelAuthenticationChallenge;
   __block NSURLCredential *credential = nil;
 
-  NSString *pemCert = StatusgoImageServerTLSCert();
+  NSString *pemCert = [StatusBackendClient executeStatusGoRequestWithResult:@"ImageServerTLSCert" 
+                                                             body:@""
+                                                 statusgoFunction:^NSString *{
+        return StatusgoImageServerTLSCert();
+    }];
+
   pemCert = [pemCert stringByReplacingOccurrencesOfString:@"-----BEGIN CERTIFICATE-----\n" withString:@""];
   pemCert = [pemCert stringByReplacingOccurrencesOfString:@"\n-----END CERTIFICATE-----" withString:@""];
   NSData *derCert = [[NSData alloc] initWithBase64EncodedString:pemCert options:NSDataBase64DecodingIgnoreUnknownCharacters];

modules/react-native-status/android/src/main/java/im/status/ethereum/module/AccountManager.kt

@@ -24,25 +24,23 @@ class AccountManager(private val reactContext: ReactApplicationContext) : ReactC
     @ReactMethod
     fun createAccountAndLogin(createAccountRequest: String) {
         Log.d(TAG, "createAccountAndLogin")
-        val result = Statusgo.createAccountAndLogin(createAccountRequest)
-        if (result.startsWith("{\"error\":\"\"")) {
-            Log.d(TAG, "createAccountAndLogin success: $result")
-            Log.d(TAG, "Geth node started")
-        } else {
-            Log.e(TAG, "createAccountAndLogin failed: $result")
-        }
+
+        StatusBackendClient.executeStatusGoRequest(
+            endpoint = "CreateAccountAndLogin",
+            requestBody = createAccountRequest,
+            statusgoFunction = { Statusgo.createAccountAndLogin(createAccountRequest) }
+        )
     }
 
     @ReactMethod
     fun restoreAccountAndLogin(restoreAccountRequest: String) {
         Log.d(TAG, "restoreAccountAndLogin")
-        val result = Statusgo.restoreAccountAndLogin(restoreAccountRequest)
-        if (result.startsWith("{\"error\":\"\"")) {
-            Log.d(TAG, "restoreAccountAndLogin success: $result")
-            Log.d(TAG, "Geth node started")
-        } else {
-            Log.e(TAG, "restoreAccountAndLogin failed: $result")
-        }
+
+        StatusBackendClient.executeStatusGoRequest(
+            endpoint = "RestoreAccountAndLogin",
+            requestBody = restoreAccountRequest,
+            statusgoFunction = { Statusgo.restoreAccountAndLogin(restoreAccountRequest) }
+        )
     }
 
     private fun updateConfig(jsonConfigString: String, absRootDirPath: String, keystoreDirPath: String): String {
@@ -173,12 +171,7 @@ class AccountManager(private val reactContext: ReactApplicationContext) : ReactC
                 accountsData,
                 chatKey
             )
-            if (result.startsWith("{\"error\":\"\"")) {
-                Log.d(TAG, "saveAccountAndLoginWithKeycard result: $result")
-                Log.d(TAG, "Geth node started")
-            } else {
-                Log.e(TAG, "saveAccountAndLoginWithKeycard failed: $result")
-            }
+            utils.handleStatusGoResponse(result, "saveAccountAndLoginWithKeycard")
         } catch (e: JSONException) {
             Log.e(TAG, "JSON conversion failed: ${e.message}")
         }
@@ -189,34 +182,25 @@ class AccountManager(private val reactContext: ReactApplicationContext) : ReactC
         Log.d(TAG, "loginWithKeycard")
         utils.migrateKeyStoreDir(accountData, password)
         val result = Statusgo.loginWithKeycard(accountData, password, chatKey, nodeConfigJSON)
-        if (result.startsWith("{\"error\":\"\"")) {
-            Log.d(TAG, "LoginWithKeycard result: $result")
-        } else {
-            Log.e(TAG, "LoginWithKeycard failed: $result")
-        }
+        utils.handleStatusGoResponse(result, "loginWithKeycard")
     }
 
     @ReactMethod
     fun loginWithConfig(accountData: String, password: String, configJSON: String) {
         Log.d(TAG, "loginWithConfig")
         utils.migrateKeyStoreDir(accountData, password)
         val result = Statusgo.loginWithConfig(accountData, password, configJSON)
-        if (result.startsWith("{\"error\":\"\"")) {
-            Log.d(TAG, "LoginWithConfig result: $result")
-        } else {
-            Log.e(TAG, "LoginWithConfig failed: $result")
-        }
+        utils.handleStatusGoResponse(result, "loginWithConfig")
     }
 
     @ReactMethod
     fun loginAccount(request: String) {
         Log.d(TAG, "loginAccount")
-        val result = Statusgo.loginAccount(request)
-        if (result.startsWith("{\"error\":\"\"")) {
-            Log.d(TAG, "loginAccount result: $result")
-        } else {
-            Log.e(TAG, "loginAccount failed: $result")
-        }
+        StatusBackendClient.executeStatusGoRequest(
+            endpoint = "LoginAccount",
+            requestBody = request,
+            statusgoFunction = { Statusgo.loginAccount(request) }
+        )
     }
 
     @ReactMethod
@@ -229,8 +213,10 @@ class AccountManager(private val reactContext: ReactApplicationContext) : ReactC
         jsonParams.put("address", address) 
         jsonParams.put("password", password)
 
-        utils.executeRunnableStatusGoMethod(
-            { Statusgo.verifyAccountPasswordV2(jsonParams.toString()) },
+        StatusBackendClient.executeStatusGoRequestWithCallback(
+            endpoint = "VerifyAccountPasswordV2",
+            requestBody = jsonParams.toString(),
+            statusgoFunction = { Statusgo.verifyAccountPasswordV2(jsonParams.toString()) },
             callback
         )
     }
@@ -241,8 +227,12 @@ class AccountManager(private val reactContext: ReactApplicationContext) : ReactC
         jsonParams.put("keyUID", keyUID)
         jsonParams.put("password", password)
 
-        utils.executeRunnableStatusGoMethod(
-            { Statusgo.verifyDatabasePasswordV2(jsonParams.toString()) },
+        val jsonString = jsonParams.toString()
+
+        StatusBackendClient.executeStatusGoRequestWithCallback(
+            endpoint = "VerifyDatabasePasswordV2",
+            requestBody = jsonString,
+            statusgoFunction = { Statusgo.verifyDatabasePasswordV2(jsonString) },
             callback
         )
     }
@@ -258,63 +248,103 @@ class AccountManager(private val reactContext: ReactApplicationContext) : ReactC
     @ReactMethod
     private fun initializeApplication(request: String, callback: Callback) {
         Log.d(TAG, "initializeApplication")
-        Log.d(TAG, "[Initializing application $request")
-        utils.executeRunnableStatusGoMethod({ Statusgo.initializeApplication(request) }, callback)
+        StatusBackendClient.executeStatusGoRequestWithCallback(
+            "InitializeApplication",
+            request,
+            { Statusgo.initializeApplication(request) },
+            callback
+        )
     }
 
     @ReactMethod
     private fun acceptTerms(callback: Callback) {
         Log.d(TAG, "acceptTerms")
-        utils.executeRunnableStatusGoMethod({ Statusgo.acceptTerms() }, callback)
+        StatusBackendClient.executeStatusGoRequestWithCallback(
+            "AcceptTerms",
+            "",
+            { Statusgo.acceptTerms() },
+            callback
+        )
     }
 
     @ReactMethod
     fun logout() {
         Log.d(TAG, "logout")
-        val runnable = Runnable {
-            val result = Statusgo.logout()
-            if (result.startsWith("{\"error\":\"\"")) {
-                Log.d(TAG, "Logout result: $result")
-            } else {
-                Log.e(TAG, "Logout failed: $result")
-            }
-        }
-        StatusThreadPoolExecutor.getInstance().execute(runnable)
+        StatusBackendClient.executeStatusGoRequest(
+            endpoint = "Logout",
+            requestBody = "",
+            statusgoFunction = { Statusgo.logout() }
+        )
     }
 
     @ReactMethod
     fun multiAccountStoreAccount(json: String, callback: Callback) {
-        utils.executeRunnableStatusGoMethod({ Statusgo.multiAccountStoreAccount(json) }, callback)
+        StatusBackendClient.executeStatusGoRequestWithCallback(
+            endpoint = "MultiAccountStoreAccount",
+            requestBody = json,
+            statusgoFunction = { Statusgo.multiAccountStoreAccount(json) },
+            callback = callback
+        )
     }
 
     @ReactMethod
     fun multiAccountLoadAccount(json: String, callback: Callback) {
-        utils.executeRunnableStatusGoMethod({ Statusgo.multiAccountLoadAccount(json) }, callback)
+        StatusBackendClient.executeStatusGoRequestWithCallback(
+            endpoint = "MultiAccountLoadAccount",
+            requestBody = json,
+            statusgoFunction = { Statusgo.multiAccountLoadAccount(json) },
+            callback = callback
+        )
     }
 
     @ReactMethod
     fun multiAccountDeriveAddresses(json: String, callback: Callback) {
-        utils.executeRunnableStatusGoMethod({ Statusgo.multiAccountDeriveAddresses(json) }, callback)
+        StatusBackendClient.executeStatusGoRequestWithCallback(
+            endpoint = "MultiAccountDeriveAddresses",
+            requestBody = json,
+            statusgoFunction = { Statusgo.multiAccountDeriveAddresses(json) },
+            callback = callback
+        )
     }
 
     @ReactMethod
     fun multiAccountGenerateAndDeriveAddresses(json: String, callback: Callback) {
-        utils.executeRunnableStatusGoMethod({ Statusgo.multiAccountGenerateAndDeriveAddresses(json) }, callback)
+        StatusBackendClient.executeStatusGoRequestWithCallback(
+            endpoint = "MultiAccountGenerateAndDeriveAddresses",
+            requestBody = json,
+            statusgoFunction = { Statusgo.multiAccountGenerateAndDeriveAddresses(json) },
+            callback = callback
+        )
     }
 
     @ReactMethod
     fun multiAccountStoreDerived(json: String, callback: Callback) {
-        utils.executeRunnableStatusGoMethod({ Statusgo.multiAccountStoreDerivedAccounts(json) }, callback)
+        StatusBackendClient.executeStatusGoRequestWithCallback(
+            endpoint = "MultiAccountStoreDerivedAccounts",
+            requestBody = json,
+            statusgoFunction = { Statusgo.multiAccountStoreDerivedAccounts(json) },
+            callback = callback
+        )
     }
 
     @ReactMethod
     fun multiAccountImportMnemonic(json: String, callback: Callback) {
-        utils.executeRunnableStatusGoMethod({ Statusgo.multiAccountImportMnemonic(json) }, callback)
+        StatusBackendClient.executeStatusGoRequestWithCallback(
+            endpoint = "MultiAccountImportMnemonic",
+            requestBody = json,
+            statusgoFunction = { Statusgo.multiAccountImportMnemonic(json) },
+            callback = callback
+        )
     }
 
     @ReactMethod
     fun multiAccountImportPrivateKey(json: String, callback: Callback) {
-        utils.executeRunnableStatusGoMethod({ Statusgo.multiAccountImportPrivateKey(json) }, callback)
+        StatusBackendClient.executeStatusGoRequestWithCallback(
+            endpoint = "MultiAccountImportPrivateKey",
+            requestBody = json,
+            statusgoFunction = { Statusgo.multiAccountImportPrivateKey(json) },
+            callback = callback
+        )
     }
 
     @ReactMethod
@@ -325,25 +355,42 @@ class AccountManager(private val reactContext: ReactApplicationContext) : ReactC
             put("keyStoreDir", keyStoreDir)
         }
         val jsonString = params.toString()
-        utils.executeRunnableStatusGoMethod({ Statusgo.deleteMultiaccountV2(jsonString) }, callback)
+        StatusBackendClient.executeStatusGoRequestWithCallback(
+            endpoint = "DeleteMultiaccountV2",
+            requestBody = jsonString,
+            statusgoFunction = { Statusgo.deleteMultiaccountV2(jsonString) },
+            callback
+        )
     }
 
     @ReactMethod
     fun getRandomMnemonic(callback: Callback) {
-        utils.executeRunnableStatusGoMethod({ Statusgo.getRandomMnemonic() }, callback)
+        StatusBackendClient.executeStatusGoRequestWithCallback(
+            "GetRandomMnemonic",
+            "",
+            { Statusgo.getRandomMnemonic() },
+            callback
+        )
     }
 
     @ReactMethod
     fun createAccountFromMnemonicAndDeriveAccountsForPaths(mnemonic: String, callback: Callback) {
-        utils.executeRunnableStatusGoMethod(
+        StatusBackendClient.executeStatusGoRequestWithCallback(
+            "CreateAccountFromMnemonicAndDeriveAccountsForPaths",
+            mnemonic,
             { Statusgo.createAccountFromMnemonicAndDeriveAccountsForPaths(mnemonic) },
             callback
         )
     }
 
     @ReactMethod
     fun createAccountFromPrivateKey(json: String, callback: Callback) {
-        utils.executeRunnableStatusGoMethod({ Statusgo.createAccountFromPrivateKey(json) }, callback)
+        StatusBackendClient.executeStatusGoRequestWithCallback(
+            endpoint = "CreateAccountFromPrivateKey",
+            requestBody = json,
+            statusgoFunction = { Statusgo.createAccountFromPrivateKey(json) },
+            callback = callback
+        )
     }
 
     companion object {