Support building go-app with GopherJS (no wasm)

[akolybelnikov]

Currently go-app assumes that the target architecture for the client side must be GOOS=js GOARCH=wasm. Here is a POC of the Lofimusic app build with GopherJS with a minor patch to the go-app:
https://github.com/nevkontakte/lofimusic/tree/0d328bf50b6fe7a3fc5f1a5d0d06a99f3e7cc368

The compiled code for GopherJS looks very different from Go-wasm, but it supports all the same APIs, including "syscall/js". The only major difference between Go-wasm and GopherJS is how the code is initialized: GopherJS produces a self-sufficient JS script, so it only needs to be included in the page via a <script> tag.

If as a go-app user you want to make a choice at the build time whether you want to use GopherJS or wasm, the framework could generate the appropriate initialization scripts.

As an aside, although GopherJS does support syscall/js API, it is a wrapper around the github.com/gopherjs/gopherjs/js package and is a bit slower than using the latter directly. It should be easy to make a GopherJS-specific implementation of that for GOOS=js GOARCH=ecmascrit target, thus gaining some performance.

[prologic]

What's the advantage of doing this besides the bundled JS that can be included directly in the page?

[oderwat]

GopherJS has 32 bit int, which would give some of my apps problems. But I am also not sure how this will work when using GopherJS because afaik the go-app frontend code is running in another thread (from the service worker). I wonder how well GopherJS handles concurrency. You write it gets loaded in the page, wouldn't this be inside the UI routine of the browser. I may not understand enough about that topic though, but I read that GopherJS has quite some overhead for channels (because of blocking code not being possible in the main browser js thread).

[oderwat]

JS that can be included directly in the page?

You could bundle the WASM too but the whole PWA aspect is about using the fetch cache interface of the web worker so it does not get loaded again and again, as it would be if you embed it into the html for the site.

[prologic]

Right now I just don't see how GopherJS and WASM are even compatible 🤔

[maxence-charriere]

To be honest folks, gopherjs might be interesting but it is Go transpilled to js and there might be compatibility issue.

For me the problem is a bit like with tinygo. Maintaining it with Go official support is already a lot of work and I don’t have the time and resources to add tinygo or gopherjs on the top of that 😅.

[nevkontakte]

Hi folks, gopherjs maintainer here 🙂

Let me be a bit more background to this request, and maybe a reason to reconsider the decision. GopherJS has a playground similar to the one Go itself offers. One of its features is that it is compiled by gopherjs itself, which allows it to run fully client-side, in pretty much any browser. It serves a double-duty as one of our reference apps we use to test gopherjs against.

The playground though is due an overhaul. Currently it's based on a very old and very unsupported version of AngularJS. So before we begin adding features we wanted to rewrite the playground on a more modern and actively evolving framework. And it seemed elegant to pick one of the Go-based options instead of making bindings to something like React. @akolybelnikov suggested go-app as an option except that, of course, it currently only targets wasm.

To test feasibility of the idea I threw together a quick proof-of-concept, https://github.com/maxence-charriere/lofimusic seemed like a complex enough app to catch most issues. Turns out, there is very little that needs to change in go-app to make it work with gopherjs. This commit is a quick-and-dirty demonstration of that:

• In the go-app code itself it only replaces checks for GOOS=wasm with GOOS=js (which would match both Go WebAssembly and GopherJS).
• pkg/app/gen/app.js instead of loading the wasm blob loads the GopherJS-generated JavaScript.

This all works because gopherjs provides full support for the standard syscall/js package, and we go to some length to keep working with each Go release. Of course, to do this properly we'd probably have to create a different version of the pkg/app/gen/app.js script that doesn't bring in wasm-related stuff. And when a new go-app project is created, the users would be able to choose whether to use Go Wasm or GopherJS.

So I think being compatible with gopherjs won't put too much burden on you, most of it is born by gopherjs being compatible with Go Wasm.

In theory, some performance could be gained by writing a gopherjs-specific version of https://github.com/maxence-charriere/go-app/blob/master/pkg/app/js_wasm.go using the github.com/gopherjs/gopherjs/js package instead of syscall/js, but that's totally optional.

As a side note, the lofimusic app compiled with Go Wasm is 3.3M (gzipped) and GopherJS is 1.7M (also gzipped). That can be a significant appeal in cases where reducing asset size is a concern.

Please let me know what you think 🙂

[oderwat]

Well, we have some much larger and more complex applications than lofimusic. It would be interesting to check out if they work. One public proof of concept is at https://github.com/oderwat/go-nats-app. Do you care to create a PR to make this compile with GopherJS? If that works, I may try out some of our larger applications that are a lot bigger than this demonstration.

P.S.: Going JS looks backwards to me. I would prefer more (Go) people would work on WASM, which has quite some interesting feats for the future.

[maxence-charriere]

I ll have another look and evaluate this.

[nevkontakte]

@oderwat here you go: https://github.com/nevkontakte/go-nats-app.

If you want to play around with my fork locally you need to install gopherjs from the latest master: go install github.com/gopherjs/gopherjs@master. It includes a couple of standard library fixes that make it work in an environment where wasm_exec.js is loaded. Same caveats apply, of course: it uses my quick-and-dirty fork of go-app, not something I would recommend for production use just yet.

P.S.: Going JS looks backwards to me. I would prefer more (Go) people would work on WASM, which has quite some interesting feats for the future.

I get that a lot, which reminds me every time that GopherJS has ways to go with regards to explaining its purpose and general PR. In short, compiling to JS has its own unique benefits, for example interop with native JS code (which is still going to be lingua franca of web frontends for years to come), broader browser support. It also used to outperform Go Wasm, but the most recent benchmarks I know of are from before I got involved with the project, so I don't know if this is still the case. This is the same kind of question as "why Go supports risc-v architecture, arm is clearly superior" ¯_(ツ)_/¯

[oderwat]

@nevkontakte very cool. I will have a look at this.

I hope we can agree that it is fine to use what is "mainstream" and still be able to see the flaws and work on replacing it with something that is suited better for the task at hand. Likewise, I look forward to more projects like Scale and debugging, real threading and network access for WASM in the browser.

P.S.: I guess you meant "why go supports arm and not risc-v". Well, I guess it supports what is being running on the majority of computers we use.

[oderwat]

@nevkontakte I checked out the GopherJS and I am impressed. I hope I get some time to check this with some other projects of us. It is fascinating that it works and even when the uncompressed code seems to have a similar size (8-9 MB), the gzip version of the JS variant is about 50% of the WASM version. Definitely something that makes sense in production.

I have a complex app that runs a bit sluggish on old Android devices. I guess this will be the first target for me to test in a GopherJS version.

[nevkontakte]

@oderwat I'm getting off topic, but Go actually does support risc-v, specifically GOOS=linux GOARCH=riscv64 :) I guess the idea I was really trying to communicate is that I regard EcmaScript as just another low-level architecture Go could be compiled to. It's a niche one for sure, but it has use cases where it is the best tool. Likewise, wasm has applications in many other situations where it is best (for example, if you needed to do number-crunching like custom cryptography or neural networks, wasm would be a better choice than JS). So I think we are on the same page, and I will wait for @maxence-charriere to weigh in 🙂

[nevkontakte]

@maxence-charriere have you had a chance to consider this request? I'll have some spare time next weekend, which I could spend putting together a proper PR.

[oderwat]

@nevkontakte Sorry, but I forgot to report back from trying to build one of our larger projects.

Here is the short version of the journey:

I had to basically delete "../../../../../../sdk/go1.18.6/src/net/http/pprof/pprof.go" because of

../../../../../../sdk/go1.18.6/src/net/http/pprof/pprof.go:380:26: undefined: pprof.Profiles
Error: running "gopherjs build -o ./web/app.js ./pwa" failed with exit code 1

Then I got an error about the missing runtime/debug.modinfo

Next I had Go 1.18 vs Go 1.120 related problems:

• It mourned bad embed statement locations that are fine with 1.20. I could fix that thought.
• We use errors.Join() in nearly all packages that were refactored lately. I went through removing them all for this test.

Finally, I ended up with another problem in our AvroX package:

function UnmarshalAny: type parameters are not supported by GopherJS:

This makes all of our PWAs impossible to compile with GopherJS as we use AvroX quite everywhere. But I just removed all of it from one of the PWAs because I was curious what happens next.

Which then was the int32 problem (as I stated at the very beginning of this issue)

8_000_000_000_000 (untyped int constant 8000000000000) overflows int

So, I removed the heart of that App's functionality and continued to compile. It then compiled, and I could try to run the app. While it initializes, it throws:

app.js:5 Uncaught Error: runtime error: index out of range
    at $callDeferred (app.js:5:25150)
    at $panic (app.js:5:25733)
    at AU (gopherjs__runtime.go:486:3)
    at Object.AG [as FuncForPC] (gopherjs__runtime.go:428:3)
    at Object.E [as Log] (dbg.go:36:3)
    at Object.I [as Create] (frontend.go:13:3)
    at Object.G [as Create] (ux.go:51:3)
    at E (skelly_ux.go:18:3)
    at F (main.go:21:3)
    at $init (app.js:172:1618)
    at $goroutine (app.js:5:26258)
    at $runScheduled (app.js:5:26982)
    at $schedule (app.js:5:27186)
    at $go (app.js:5:26827)
    at app.js:177:1
    at app.js:180:4

I think that is related to using the runtime for finding the call-stack for our dbg package. I could go and fix that too. But I stopped here and changed back to the WASM backend and compiled the same completely crippled code, and it just started as usual (minus the actual functionalities).

I fear we most likely will never be able to use GopherJS with Go-App in anything that goes into production. We do use GopherJS in that same project for creating some (rather toy examples) of frontend JavaScript to handle some buttons, and I was impressed how it magically works. But there it adds a ton of JS code for something that needs 10 lines of native JS.

Some other consideration at the end: The crippled app compiled into 8.4M + 487K (map) for JS and 9.1M for WASM. I don't see that this would make us consider using JS instead of WASM, either. Even if the full working app currently compiles to 16M, I doubt that having all the code compiling to JS will shave off enough size, that a rewrite of all the problematic parts would be feasible.

[nevkontakte]

@oderwat thanks for the report, this is interesting to know. Let me give a brief response, and I'll be happy to have a more detailed discussion with you elsewhere (feel free to file an issue in the GopherJS issue tracker or ping me in the Gophers Slack)

• Go 1.20 and generics support. This is coming - eventually. For the last half year I've been working on the generics support, and at this point I can see the light at the end of the tunnel. Unfortunately, this turned out a really complex project and I'm the only one working on it, so it's not fast... Once I'm done with generics, I'll be catching up on Go releases.
• net/http/pprof is one of the few packages we don't support properly - for historical reasons. I'm pretty sure it should be possible at least make a compilable no-op implementation, but nobody complained about it, so it has been pretty low on my priority list.
• The int32 problem - arguably if you are doing 64-bit math, you should use int64. While GopherJS behaves as a 32-bit architecture for performance reasons, it does have full support for int64, even though it is somewhat slower than native.
• The index out of range error - this might be a bug, but without seeing the code it's hard for me to tell more. If you could file an issue in our issue tracker with a code sample, that would be very helpful.

[nevkontakte]

@maxence-charriere even if you don't want to support GopherJS officially, I think it is possible to make a few changes that will make it at least not actively incompatible, and the interested users would be able to inject the required initialization scripts themselves. That will be sufficient for our purposes, and won't put any additional burden on you.

[maxence-charriere]

I ll take a look on this by the end of the month.

[gedw99]

This is really interesting stuff…

I am surprise that gopherjs can do this.

i think that it’s best to give more time for evaluation of gopherjs.

Its maybe just me but I feel that it’s worthwhile to let things evolve a bit … I mean it’s amazing it works and or course there will be some rough edges.

On a side note. I don’t know if anyone has considered this but there might be advantages of even having the PWA service worker in wasm and the DOM Renderer in gopherjs. There are a few use cases where separating the two is useful from a concurrency / threading point of view.

my own worry is that gopherjs has only a few maintainers ? whereas golang wasm is more supported by the golang team. Tinygo is also financially supported AFAIK by google.

[gedw99]

@nevkontakte Have you considered the possibility of maintaining any gopherjs compile issues in go-app ?

I only ask because @maxence-charriere would be having to support 2 compiler paths which is more work !!

[maxence-charriere]

I was not able to use gopherjs build since it requires an older version of Go.

What I'm thinking is to introduce a Driver interface that would load either the default wasm, a gopherjs or tinygo implementation. This would open the field to other options without me having to maintain them.

It is a bit unclear to me what are gopherjs requirements.
It seems that gopherjs build generates a .js file that is the equivalent of a .wasm file.

I have a few questions:

• Does gopher js require including another js file such as wasm_exec.js?
• When does gopher.js app main is started? Can it be manually run with a js command?

[oderwat]

@maxence-charriere to get the right gopherjs you need to install it like this:

# install the correct GopherJS
go install github.com/gopherjs/gopherjs@master
go install golang.org/dl/go1.18.6@latest
#  clone the converted go-nats example
git clone https://github.com/nevkontakte/go-nats-app go-nats-app-gjs
# enter the directory
cd go-nats-app-gjs
# build and run it
go mage.go run

Notice the replaced Go-App dependency in the go.mod file, that uses his go-app fork with some (actually minor) changes: master...nevkontakte:go-app:master

From what I see, you need to switch the build tags and then just need to load the GopherJS app.js file instead of the WASM and this is started automatically when the browser finished loading it. I wonder if that could be changed with a flag when compiling the code.

In theory, some performance could be gained by writing a gopherjs-specific version of https://github.com/maxence-charriere/go-app/blob/master/pkg/app/js_wasm.go using the github.com/gopherjs/gopherjs/js package instead of syscall/js, but that's totally optional.

So, he is using your version right now, but that could be probably optimized for GopherJS.

[oderwat]

@nevkontakte I guess for us, it comes down to generics support and newer Go compilers. This is what makes rewriting the code impossible for us. And you are right: We should use int64 in the parts that really need int64, I actually consider rewriting it (for other reasons too). I am not sure what the net/http/pprof package is doing there, and the panic was related to runtime.Callers() / runtime.FuncForPC which could be removed, or better, replaced with a way to get the same information in a GopherJS way.

[gedw99]

@nevkontakte mentioned that the profiling can be made to be agnostic based on build time or runtime tags.

it’s mentioned higher up in this issue ..

“

net/http/pprof

few packages we don't support properly - for historical reasons. I'm pretty sure it should be possible at least make a compilable no-op implementation, but nobody complained about it, so it has been pretty low on my priority list.

“
might be a good solution. I have had to do a similar thing before

[nevkontakte]

Wow, a lot to respond to 🙂

my own worry is that gopherjs has only a few maintainers ?

That is correct. Besides myself, there's another person who (for real life reasons) at the moment can only really provide code reviews, but doesn't have enough time to contribute code. I would happily onboard more people, but it's been hard to find volunteers so far. Also, gopherjs is not my full-time job, so it gets much less time than it deserves. Which leads me to...

Have you considered the possibility of maintaining any gopherjs compile issues in go-app ?

I have, and I'm somewhat on a fence about this. On one hand, I don't think that would be that much work, and it's a nice change of pace from developing gopherjs itself. On the other hand, I don't want to give promises I am unsure I'll be able to keep. And gopherjs itself takes pretty much all the time I have for pet projects.

I only ask because @maxence-charriere would be having to support 2 compiler paths which is more work !!

So the theory goes that any program that works under wasm and doesn't use unsafe should work under gopherjs. There are three places where the theory doesn't match reality:

1. GopherJS uses GOOS=js GOARCH=ecmascript build tags, but many Go Wasm programs use //go:build wasm constraint, which is more restrictive than what they actually need. //go:build js would be sufficient for most of them and wouldn't actively prevent GopherJS from working.
2. Runtime initialization is different: Go Wasm requires wasm_exec.js and the usual "load and run wasm blob" prelude, whereas GopherJS generates a self-sufficient JS file, which can be directly included via the <script src="..."> tag.
3. There are some standard library packages that GopherJS doesn't support. Usually they have to do with the low-level runtime stuff like unsafe and pprof. With each release I'm trying to close that gap a bit more, but obviously pointer arithmetics is just impossible to do in JS :)

My thinking was that a few simple changes in go-app API can address points #1 and #2. For #2 specifically it could provide an option for the alternate initialization script, in the same way it already does for the service worker, and with the same warning. Then for @maxence-charriere the support story doesn't change - wasm would be the main targeted architecture, but those brave enough to tinker can use other toolchains - and enjoy the debugging :)

I think at this point it would be more productive for me to actually put together a strawman PR instead of handwaving, and then we could have this discussion more concretely.

Does gopher js require including another js file such as wasm_exec.js?

No, its output is self-sufficient and can be included directly in the page with a <script src="..."> tag.

When does gopher.js app main is started? Can it be manually run with a js command?

main() starts executing synchronously whenever the JS file is evaluated. If you want to delay that - you could construct the <script> tag at runtime, though nobody has requested that so far.

the panic was related to runtime.Callers() / runtime.FuncForPC which could be removed, or better, replaced with a way to get the same information in a GopherJS way.

GopherJS should support those functions, with the limitation that runtime.FuncForPC() can be only called with the pc that was previously returned by one of the other runtime functions, without any modifications. So, if you can provide a way to reproduce the panic, this seems like something we should fix.

[gedw99]

@nevkontakte strawman PR 👍 Like you said it will help move things forward with less unknowns.

I am really interested in leveraging both in the one app with wasm for the Service Worker and Web Workers and JS for the Rendering. Its is the best use of both IMHO and could lead to higher perf.

I have not looked into the BUS in go-app between the SW and DOM layers. I assume there is an abstracted one in the code base @maxence-charriere ?

[nevkontakte]

I guess for us, it comes down to generics support and newer Go compilers.

@oderwat then the good news is - both are coming. I hope to finish work on generics within the next a couple of months, then catch up on Go releases.

[nevkontakte]

As promised, #830 is a cleaner way of using go-app with gopherjs. I took @maxence-charriere idea of implementing a Driver interface that's responsible for setting up the client-side and moved wasm-specific bits inside it.

Let me know if that looks reasonable to y'all.

[oderwat]

I can still compile to WASM with that branch and no changes to my pretty complex setup, makes me happy ;-)