Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feat/use status backend server #21550

Merged
merged 4 commits into from
Nov 14, 2024

Conversation

qfrank
Copy link
Contributor

@qfrank qfrank commented Nov 4, 2024

Finally, we can now use the status backend server in the local development environment with this pull request without having to wait for the painfully long rebuild time! I created this PR to facilitate review. Once it is reviewed, we can merge it and conduct a manual QA on PR 21450.

Pls see the doc for usage.

@status-im-auto
Copy link
Member

status-im-auto commented Nov 4, 2024

Jenkins Builds

Click to see older builds (65)
Commit #️⃣ Finished (UTC) Duration Platform Result
✔️ 2e8449c #1 2024-11-04 04:44:44 ~5 min tests 📄log
✔️ 2e8449c #1 2024-11-04 04:47:40 ~8 min android 🤖apk 📲
✔️ 2e8449c #1 2024-11-04 04:47:50 ~8 min android-e2e 🤖apk 📲
✔️ 2e8449c #1 2024-11-04 04:49:48 ~10 min ios 📱ipa 📲
b1ad9c8 #2 2024-11-04 06:58:45 ~3 min tests 📄log
✔️ b1ad9c8 #2 2024-11-04 07:01:42 ~6 min android-e2e 🤖apk 📲
✔️ b1ad9c8 #2 2024-11-04 07:03:45 ~8 min android 🤖apk 📲
✔️ b1ad9c8 #2 2024-11-04 07:04:02 ~9 min ios 📱ipa 📲
5101537 #3 2024-11-04 07:09:08 ~2 min tests 📄log
✔️ 5101537 #3 2024-11-04 07:12:49 ~6 min android-e2e 🤖apk 📲
✔️ 5101537 #3 2024-11-04 07:14:37 ~8 min android 🤖apk 📲
✔️ 5101537 #3 2024-11-04 07:15:19 ~8 min ios 📱ipa 📲
0b2aa2c #4 2024-11-04 07:33:01 ~2 min tests 📄log
✔️ 0b2aa2c #4 2024-11-04 07:36:48 ~6 min android-e2e 🤖apk 📲
✔️ 0b2aa2c #4 2024-11-04 07:38:20 ~7 min android 🤖apk 📲
✔️ 0b2aa2c #4 2024-11-04 07:39:19 ~8 min ios 📱ipa 📲
6ae49a5 #5 2024-11-04 09:08:08 ~2 min tests 📄log
✔️ 6ae49a5 #5 2024-11-04 09:13:46 ~8 min android-e2e 🤖apk 📲
✔️ 6ae49a5 #5 2024-11-04 09:14:13 ~8 min android 🤖apk 📲
✔️ 6ae49a5 #5 2024-11-04 09:16:47 ~11 min ios 📱ipa 📲
376a232 #6 2024-11-05 08:12:38 ~2 min tests 📄log
✔️ 376a232 #6 2024-11-05 08:16:25 ~6 min android-e2e 🤖apk 📲
✔️ 376a232 #6 2024-11-05 08:17:00 ~6 min android 🤖apk 📲
✔️ 376a232 #6 2024-11-05 08:19:14 ~9 min ios 📱ipa 📲
✔️ 3358e8e #7 2024-11-05 09:07:47 ~4 min tests 📄log
✔️ 3358e8e #7 2024-11-05 09:10:13 ~6 min android 🤖apk 📲
✔️ 3358e8e #7 2024-11-05 09:10:52 ~7 min android-e2e 🤖apk 📲
✔️ 3358e8e #7 2024-11-05 09:12:14 ~8 min ios 📱ipa 📲
✔️ da9bec7 #8 2024-11-05 11:59:30 ~4 min tests 📄log
✔️ da9bec7 #8 2024-11-05 12:01:27 ~6 min android-e2e 🤖apk 📲
✔️ da9bec7 #8 2024-11-05 12:03:07 ~8 min android 🤖apk 📲
✔️ da9bec7 #8 2024-11-05 12:03:54 ~8 min ios 📱ipa 📲
f1654d5 #9 2024-11-06 03:06:48 ~2 min ios 📄log
f1654d5 #9 2024-11-06 03:07:10 ~2 min tests 📄log
✔️ f1654d5 #9 2024-11-06 03:11:11 ~6 min android 🤖apk 📲
✔️ f1654d5 #9 2024-11-06 03:11:42 ~7 min android-e2e 🤖apk 📲
4caa788 #10 2024-11-06 03:54:19 ~2 min ios 📄log
4caa788 #10 2024-11-06 03:54:37 ~2 min tests 📄log
✔️ 4caa788 #10 2024-11-06 03:58:35 ~6 min android 🤖apk 📲
✔️ 4caa788 #10 2024-11-06 03:59:08 ~7 min android-e2e 🤖apk 📲
0b21ea1 #11 2024-11-08 11:33:39 ~2 min ios 📄log
0b21ea1 #11 2024-11-08 11:34:03 ~2 min tests 📄log
✔️ 0b21ea1 #11 2024-11-08 11:39:42 ~8 min android-e2e 🤖apk 📲
✔️ 0b21ea1 #11 2024-11-08 11:40:12 ~8 min android 🤖apk 📲
073385b #12 2024-11-08 11:45:32 ~2 min ios 📄log
960c64d #13 2024-11-08 11:49:26 ~3 min tests 📄log
960c64d #13 2024-11-08 11:49:39 ~3 min ios 📄log
✔️ 960c64d #13 2024-11-08 11:53:01 ~7 min android-e2e 🤖apk 📲
✔️ 960c64d #13 2024-11-08 11:56:02 ~10 min android 🤖apk 📲
d19fff3 #14 2024-11-08 12:14:48 ~3 min ios 📄log
d19fff3 #14 2024-11-08 12:15:33 ~4 min tests 📄log
✔️ d19fff3 #14 2024-11-08 12:18:43 ~7 min android-e2e 🤖apk 📲
✔️ d19fff3 #14 2024-11-08 12:22:06 ~10 min android 🤖apk 📲
✔️ 4e863f0 #15 2024-11-12 04:15:13 ~4 min tests 📄log
✔️ 4e863f0 #15 2024-11-12 04:18:16 ~7 min android-e2e 🤖apk 📲
✔️ 4e863f0 #15 2024-11-12 04:19:32 ~8 min ios 📱ipa 📲
✔️ 4e863f0 #15 2024-11-12 04:21:28 ~10 min android 🤖apk 📲
✔️ 1840c42 #16 2024-11-12 04:53:16 ~4 min tests 📄log
✔️ 1840c42 #16 2024-11-12 04:56:11 ~7 min android-e2e 🤖apk 📲
✔️ 1840c42 #16 2024-11-12 04:57:31 ~8 min ios 📱ipa 📲
✔️ 1840c42 #16 2024-11-12 04:58:07 ~9 min android 🤖apk 📲
29b84a2 #17 2024-11-12 11:36:36 ~2 min tests 📄log
✔️ 29b84a2 #17 2024-11-12 11:40:26 ~6 min android-e2e 🤖apk 📲
✔️ 29b84a2 #17 2024-11-12 11:41:51 ~7 min android 🤖apk 📲
✔️ 29b84a2 #17 2024-11-12 11:42:47 ~8 min ios 📱ipa 📲
Commit #️⃣ Finished (UTC) Duration Platform Result
52be6bb #18 2024-11-12 13:58:04 ~3 min tests 📄log
✔️ 52be6bb #18 2024-11-12 14:02:42 ~7 min android 🤖apk 📲
✔️ 52be6bb #18 2024-11-12 14:03:51 ~8 min ios 📱ipa 📲
✔️ 52be6bb #18 2024-11-12 14:04:00 ~9 min android-e2e 🤖apk 📲
✔️ 737c6cb #19 2024-11-14 09:41:40 ~4 min tests 📄log
✔️ 737c6cb #19 2024-11-14 09:43:25 ~6 min android-e2e 🤖apk 📲
✔️ 737c6cb #19 2024-11-14 09:45:27 ~8 min android 🤖apk 📲
✔️ 737c6cb #19 2024-11-14 09:46:00 ~9 min ios 📱ipa 📲

@qfrank qfrank force-pushed the feat/status_backend_server branch 4 times, most recently from 0b2aa2c to 6ae49a5 Compare November 4, 2024 09:05
@vkjr vkjr self-requested a review November 4, 2024 10:36
@vkjr
Copy link
Contributor

vkjr commented Nov 4, 2024

@qfrank, this is amazing, thanks a lot!

Do you notice any performance difference when using status as a server?

Copy link
Contributor

@ilmotta ilmotta left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@qfrank, I tried the PR, and it seems to work, but because status-go is still compiled for mobile, it's hard for me to tell if everything is working or if something is falling back to the "normal" status-go.

In other words, changes in status-go code still cause a full rebuild even when STATUS_BACKEND_ENABLED is 1 whenever make run-android runs (I simulated this by setting STATUS_GO_SRC_OVERRIDE, but it would be the same if switching branches when the revision in status-go-version.json is different). Are you planning on fixing this in a follow-up PR?

Also, status-backend is running correctly, but I don't see any logs in its STDOUT, only in the make run-android output. Is it because of what you said in the description? Update the Android log store location.

My configuration during runtime seems to be correct and status-backend is running well (root-data seems correct and has all db files, keystore, etc).

{:enabled?           true,
 :status-go-endpoint "http://127.0.0.1:9050/statusgo/",
 :signal-endpoint    "ws://127.0.0.1:9050/signals",
 :root-data-dir      "<path>/status-go/root-data"}

As a suggestion, apart from the PR instructions, it would be great to have documentation in the source about how to use the status-backend. Another point worth mentioning in docs is that the developer should make sure the status-backend is checked out to a revision that's at least compatible with the revision in status-mobile/status-go-version.json.

src/status_im/core.cljs Show resolved Hide resolved
src/status_im/config.cljs Show resolved Hide resolved
@qfrank
Copy link
Contributor Author

qfrank commented Nov 5, 2024

I tried the PR, and it seems to work, but because status-go is still compiled for mobile, it's hard for me to tell if everything is working or if something is falling back to the "normal" status-go.

In other words, changes in status-go code still cause a full rebuild even when STATUS_BACKEND_ENABLED is 1 whenever make run-android runs (I simulated this by setting STATUS_GO_SRC_OVERRIDE, but it would be the same if switching branches when the revision in status-go-version.json is different). Are you planning on fixing this in a follow-up PR?

Not quite.
I had considered whether we could compile status-mobile without relying on status-go. Perhaps we can use some means to achieve this purpose, but I need to discuss it with Sid. However, I think the priority of this requirement is low because with the help of this PR, in most cases, we only need to compile status-mobile once (except for modifying or adding native implementations with Kotlin or Objective-C). If STATUS_BACKEND_ENABLED(EDIT: renamed to STATUS_BACKEND_SERVER_ENABLED now) is 1, STATUS_GO_SRC_OVERRIDE or status-go-version.json should have no impact as all invocation from native-module.core will delegate to status backend server you're running, so it actually depends on what version of status-go you're running for status backend server.

Also, status-backend is running correctly, but I don't see any logs in its STDOUT, only in the make run-android output. Is it because of what you said in the description? Update the Android log store location.

I'm sorry to hear this ... Because the status backend server is not mature enough. We can improve it later in separate PR. If you want see something output from STDOUT, you can change level to DEBUG, you can add fmt.Println here to see what signal sent to frontend. you can change here to see any new websocket connected. change here and here to see what request reached out from frontend.
Additionally, there should exist requests.log and geth.log in directory <path>/status-go/root-data in your case with latest commit. But rootDataDir value(/status-go/root-data) you provided seems strange, it should be an absolute path

As a suggestion, apart from the PR instructions, it would be great to have documentation in the source about how to use the status-backend. Another point worth mentioning in docs is that the developer should make sure the status-backend is checked out to a revision that's at least compatible with the revision in status-mobile/status-go-version.json.

Sounds great, I'll add it later @ilmotta

@qfrank
Copy link
Contributor Author

qfrank commented Nov 5, 2024

@qfrank, this is amazing, thanks a lot!

Do you notice any performance difference when using status as a server?

not yet, WDYT of this performance regarding invocation log from requests.log, pls notice the duration:

2024-11-05T12:32:37.938+0800	DEBUG	RequestLogger	call	{"method": "callPrivateRPC", "params": "[{\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"wakuext_contacts\",\"params\":[]}]", "resp": "{\"jsonrpc\":\"2.0\",\"id\":1,\"result\":null}", "duration": "152.209µs"}
2024-11-05T14:11:43.752+0800	DEBUG	RequestLogger	call	{"method": "appStateChangeV2", "params": "[{\"state\":\"active\"}]", "resp": "{\"error\":\"\"}", "duration": "612.625µs"}

I think it's okay for most cases. If we have special requirements, maybe we can invoke specific native functions that pretty stable through embed statusgo or try other optimization solution, this is another feature we can do

@qfrank qfrank force-pushed the feat/status_backend_server branch 2 times, most recently from f1654d5 to 4caa788 Compare November 6, 2024 03:51
@ilmotta
Copy link
Contributor

ilmotta commented Nov 6, 2024

I had considered whether we could compile status-mobile without relying on status-go. Perhaps we can use some means to achieve this purpose, but I need to discuss it with Sid.

Thanks @qfrank. No worries, let's leave at that and we can improve in the future. One day we may get there!

Copy link
Contributor

@ilmotta ilmotta left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@qfrank I had the pleasure to review and follow instructions successfully! I could step debug, quickly restart the server, super cool 🌟 Big thanks for writing the documentation.

  1. The problem with HTTPS and the image server is no big deal for me personally.

  2. It would be nice if we could get logs in the status-backend process output (in this other PR it was working WIP: Status backend intergation #21517), but it's not a PR blocker in any way.

  3. As I mentioned in a previous comment, the end goal one day would be to not build status-go as a lib if the flag is enabled STATUS_BACKEND_SERVER_ENABLED so that we can run clients 100% decoupled from status-go, but that seems harder and would require messing with the nix layer and some client code. Of course, not a PR blocker.

status-go may start to look more appealing and friendly ;)

doc/use-status-backend-server.md Outdated Show resolved Hide resolved
@qfrank
Copy link
Contributor Author

qfrank commented Nov 11, 2024

It would be nice if we could get logs in the status-backend process output

Currently, it goes into geth.log as initLogging works. You can use tail -f geth.log to see the output. I can create a separate PR to make the log message output to stdout as well when we use the status backend server after this PR is merged.

Copy link
Member

@Samyoul Samyoul left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👏

Copy link
Member

@seanstrom seanstrom left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Awesome stuff! ✅

Copy link
Contributor

@mohsen-ghafouri mohsen-ghafouri left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔥

@qfrank qfrank merged commit 0a21ecc into chore/use_v2_endpoints Nov 14, 2024
5 checks passed
@qfrank qfrank deleted the feat/status_backend_server branch November 14, 2024 11:06
qfrank added a commit that referenced this pull request Nov 14, 2024
status-im/status-go@3951129...399c3d5

* feat: use status backend server

* chore: add env variable STATUS_BACKEND_SERVER_IMAGE_SERVER_URI_PREFIX

* update doc

* fix_: image_server lint issue
@@ -0,0 +1,53 @@
## Solution to use Status Backend Server
Copy link
Contributor

@vkjr vkjr Nov 15, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@qfrank, I'm sorry for such a late after-merge comment. I think this doc would benefit from quick introduction - defining what is Status Backend Server, what purpose it serves and what are usecases for developers.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: DONE
Development

Successfully merging this pull request may close these issues.

8 participants