Precompiled stdlib (#768)

* compiler/driver: separate compilation for static exes

triggered when the tree shaker is off, to avoid excessive compile
times.

* build-spec: remove unnecessary cc options

this was from ancient gccs...

* std/foreign: smarter define-guard macro

it really shouldn't define a feature in separate compilation.

* build-deps

* script to build libgerbil.a

* fix option folding

* add build libgerbil step to build.sh

* compile static exes with saparete comp and libgerbil by default

* build libgerbil in CI

* remove unnecessary options from crypto-test compilation

* gxc: compile static exes by default

- -static is now implied, unless you pass -dynamoc
- -dynamic builds dynamic executables
- -full-program-optimization enables the old style (slow) compilation

* gxc: fix -dynamic

* compiler/driver: expand output-bin path

ld doesn't like ~

* make: exe is now static-exe

also adds dynamic-exe: and optimized-exe: to the zoo

* os/temporaries: get rid of mktemp

tired of meaningless linker warnings...

* tols: build dynamic executables

* build-deps

* add basic gxc exe compilation tests

* run gxc tests on CI

* fix bug

* update gxc test

* update documentation about the new status quo

static executables for the win!

* remove nonsensical sentence

* build-libgerbil: compile to .c's separately

* more separate compilation

and also fix process leaks; they need to be closed.

* prelude/core: add delete-file-or-directory

* build static exe artifacts in a tmp directory to avoid races

* build-libgerbil: clean up intermediate compilation artifacts

.github/workflows/ci.yml

@@ -57,6 +57,8 @@ jobs:
         run: ./src/build.sh stage1 final
       - name: Build Gerbil stdlib
         run: ./src/build.sh stdlib
+      - name: Build Gerbil libgerbil
+        run: ./src/build.sh libgerbil
       - name: Build Gerbil lang
         run: ./src/build.sh lang
       - name: Build Gerbil tools
@@ -65,4 +67,5 @@ jobs:
         run: |
           export GERBIL_HOME=${GITHUB_WORKSPACE}
           export PATH=${GITHUB_WORKSPACE}/bin:$PATH
+          gxtest src/gerbil/test/...
           gxtest src/std/...

.gitignore

@@ -19,4 +19,3 @@ src/tutorial/lang/build-deps
 src/tutorial/proxy/build-deps
 src/tutorial/proxy/socks-proxy
 src/tutorial/proxy/tcp-proxy
-test/

doc/guide/build.md

@@ -50,6 +50,7 @@ And we can build by invoking the script:
 ``` bash
 $ chmod +x build.ss
 $ ./build.ss
+...
 ```
 
 ## Intermediate build scripts
@@ -94,70 +95,3 @@ $ ./build.ss
 ```
 
 After the initial dependency graph generation, we can build during development by reusing the dependency graph and simply invoking ./build.ss. You only need to generate a new dependency graph if your import sets change.
-
-## Building static executables
-
-Static executables are simple to build:
-
-- the executables are specified with the static-exe: build spec in place of exe:.
-- the make invocation needs static: #t to be specified so that static compilation artifacts are built for modules.
-
-However, there is a nuance: you usually don't want to build static executables with debug introspection as this will blow the executable size significantly.
-
-Perhaps the simplest way to deal with the bloat issue is to have a separate step building the executables, while still compiling library modules with debug introspection for working in the repl.
-
-The following build script breaks the build action into two steps, one for building library modules and another for building the executables:
-
-``` scheme
-#!/usr/bin/env gxi
-
-(import :std/make)
-
-;; the library module build specification
-(def lib-build-spec
-  '("util"))
-
-(def bin-build-spec
-  '((static-exe: "hello")))
-
-;; the source directory anchor
-(def srcdir
-  (path-normalize (path-directory (this-source-file))))
-
-;; the main function of the script
-(def (main . args)
-  (match args
-    (["lib"]
-     ;; this action builds the library modules -- with static compilation artifacts
-     (make srcdir: srcdir
-           bindir: srcdir
-           optimize: #t
-           debug: 'src             ; enable debugger introspection for library modules
-           static: #t              ; generate static compilation artifacts; required!
-           prefix: "example"
-           ;; build-deps: "build-deps" ; this value is the default
-           lib-build-spec))
-
-    (["bin"]
-     ;; this action builds the static executables -- no debug introspection
-     (make srcdir: srcdir
-           bindir: srcdir
-           optimize: #t
-           debug: #f               ; no debug bloat for executables
-           static: #t              ; generate static compilation artifacts; required!
-           prefix: "example"
-           build-deps: "build-deps-bin" ; importantly, pick a file that differs from above
-           bin-build-spec))
-
-    ;; this is the default action, builds libraries and executables
-    ([]
-     (main "lib")
-     (main "bin"))))
-```
-
-Note that the `build-deps:` file is a cache that stores your project dependencies.
-In large project, an up-to-date cache can save many seconds in build times.
-When multiple projects share a same directory, they must be made to use separate
-`build-deps:` file, or the caches will clash and be ineffective.
-All but one of the projects must explicitly specify the `build-deps:` argument
-to point to its own distinct file.

doc/guide/getting-started.md

@@ -153,12 +153,12 @@ representative  for your top package namespace, like your github user id.
 
 So, let's make a simple library module:
 ```
-$ mkdir -p myproject/src/myuser
-$ cd myproject/src/myuser
+$ mkdir -p myproject
+$ cd myproject
 
 # Create a gerbil.pkg file for our project
 $ cat > gerbil.pkg <<EOF
-(package: myuser)
+(package: myproject)
 EOF
 
 $ cat > mylib.ss <<EOF
@@ -173,67 +173,53 @@ You can change this by exporting the `GERBIL_PATH` variable.
 You may also explicitly use the `-d` option;
 but then you'll have to add your libdir to `GERBIL_LOADPATH`.
 ```
-$ gxc mylib.ss
+$ gxc -O mylib.ss
 ```
 
 You now have a compiled module, which you can use in the interpreter:
 ```
 $ gxi
-> (import :myuser/mylib)
+> (import :myproject/mylib)
 > (hello "world")
 hello world
 ```
 
 Next let's make an executable:
 ```
 $ cat > mybin.ss <<EOF
-(import :myuser/mylib)
+(import ./mylib)
 (export main)
 (def (main who)
  (hello who))
 EOF
 ```
 and let's compile it and run it:
 ```
-$ gxc -exe -o mybin mybin.ss
+$ gxc -O -exe -o mybin mybin.ss
 $ ./mybin world
 hello world
 ```
 
-Note that this is a dynamically linked executable, the module has been
+Note that this is a statically linked executable.
+
+If you want a dynamically linked executable, the module will be
 compiled dynamically in the gerbil libdir and the executable is a stub
-that loads it and executes main, which means that your `GERBIL_HOME`
-(and `GERBIL_LOADPATH` if you are putting your artefacts in a different
-place, like `myproject/lib`) must be set.
+that loads it and executes main. This means that your `GERBIL_HOME`
+must be set.
 
-You can also compile a statically linked executable, which can work without
-a local gerbil environment:
+You can compile a dynamic executable with the `-dynamic` flag:
 ```
-$ gxc -static mylib.ss  # compile dependent library statically first
-$ gxc -static -exe -o mybin-static mybin.ss
-$ ./mybin-static world
-hello world
+$ gxc -dynamic -exe -o mybin mybin.ss
 ```
 
 The advantage of static executables is that they can work without a local
 Gerbil installation, which makes them suitable for binary distribution.
 They also start a little faster, as there is no dynamic module loading at runtime.
-In addition, because all dependencies from the stdlib are compiled in together, you
-can apply global declarations like `(declare (not safe))` to the whole program, which
-can result in significant performance gains. And as of `Gerbil-v0.13-DEV-50-gaf81bba`
-the compiler performs full program optimization, resulting in further performance
+And if you are willing to wait a bit for your proggram to compile, you
+can specify `-full-program-optimization` which instructs the compiler
+to perform full program optimization, resulting in further performance
 benefits.
 
-The downside is long compilation times and the limitation that the executable
-won't be able to use the expander or the compiler, as the meta parts of the Gerbil
-runtime are not linked in.
-
-Note that when creating static executables, you will need to pass on options to
-the linker if you're relying on foreign libraries. For example, to
-include a dependency on `zlib`:
-```
-$ gxc -static -exe -o mybin-static -ld-options -lz mybin.ss
-```
-
-The `-ld-options` are being passed on to `gsc` which in turn adds the
-specified options to the command that invokes the C linker.
+The downside of static executables, is that they currently can't use
+the gerbil expander and compiler packages. This restriction will be
+lifted in a future release.

doc/guide/intro.md

@@ -40,24 +40,21 @@ hello world
 
 The "hello world" executable:
 ```
-$ cat > hello.ss <<EOF
-package: example
+$ cat > gerbil.pkg <<EOF
+(package: example)
+EOF
 
+$ cat > hello.ss <<EOF
 (export main)
 (def (main . args)
   (displayln "hello world"))
 EOF
-$ gxc -exe -o hello hello.ss
-$ ./hello
-hello world
-```
 
-And if you want a statically linked hello with no runtime dependencies to the Gerbil environment:
-```
-$ gxc -static -exe -o hello hello.ss
+$ gxc -O -exe -o hello hello.ss
 $ ./hello
 hello world
 ```
+
 ## Core Gerbil
 ### Primitive forms
 The standard Scheme primitive forms and macros are all supported.
@@ -769,8 +766,11 @@ to work.
 For example, suppose we have a module example/hello.ss that we
 want to compile as an executable module:
 ```
+$ cat > example/gerbil.pkg <<EOF
+(package: example)
+EOF
+
 $ cat > example/hello.ss <<EOF
-package: example
 (export main)
 (def (main . args)
   (displayln "hello world"))
@@ -783,34 +783,47 @@ runtime and module dependencies.
 You can compile it to an executable with `gxc` with the
 following command:
 ```
-$ gxc -exe -o hello example/hello.ss
+$ gxc -O -exe -o hello example/hello.ss
+$ ./hello
+hello world
+```
+
+If you want the compiler to perform full program optimization, then you can
+specify the `-full-program-optimization` flag:
+```
+$ gxc -O -full-program-optimization -exe -o hello example/hello.ss
 $ ./hello
 hello world
 ```
 
-You can also compile the executable with static linkage, which links
-parts of the gerbil runtime and all dependent modules statically and
-allows the executable to work without a local gerbil environment:
+You can also compile the executable with dynamic linkage, which requires
+a local gerbil installation at runtime.
 ```
-$ gxc -static -exe -o hello example/hello.ss
+$ gxc -O -dynamic -exe -o hello example/hello.ss
 $ ./hello
 hello world
 ```
 
-The disadvantage of static linkage is that the executables are bigger,
-and can only use the baseline parts of the gerbil runtime (gx-gambc0).
-That means that static executables can't use the expander or the compiler,
-as the meta levels of the gerbil runtime are not linked and initialized.
-They also take quite longer to compile.
-
-The advantage is that static executables don't require a local Gerbil
-installation to work, which makes them suitable for binary distribution.
-They also load faster, as they don't have to do any dynamic module loading.
-Furthermore, because dependencies are compiled in together, you can apply
-declarations like `(not safe)` to the whole program, resulting in potentially
-significant performance gains. And as of `Gerbil-v0.13-DEV-50-gaf81bba` the
-compiler performs full program optimization with tree shaking, which provides
-further performance benefits.
+The difference between the 3 executable compilation modes can be summarized as follows:
+- By default, a statically linked executable is generated, linking to the precompiled
+  gerbil standard library. Note that the executable may some have dynamic library
+  dependencies from stdlib foreign code , and also links to `libgambit`.
+  If you have configured your   gambit with `--enable-shared`, then this will be
+  a dynamic library dependency.
+- When `-full-program-optimization` is passed to `gxc`, then the compiler will perform
+  full program optimization with all gerbil library dependencies. This will result
+  both in smaller executable size and better performance, albeit at the cost of
+  increased compilation time; this can be minutes for complex programs, while
+  separately linked executables compile in a second. Furthermore, because
+  dependencies are compiled in together, you can apply declarations like `(not safe)`
+  to the whole program using the `-prelude` directive. This can result
+  in potentially significant performance gains at the expense of safety.
+- When `-dynamic` is passed to `gxc`, then a dynamic executable stub will be generated,
+  which will depend on the Gerbil runtime environment being present at execution time.
+  Dynamic executables do have some advantages over static executables however:
+  - they compile instantly and are tiny
+  - they can use the expander and the compiler; note that this restriction will be
+    lifted from static executables in a future release.
 
 ### Prelude Modules and Custom Languages
 

src/build.sh

@@ -167,6 +167,14 @@ build_stdlib () {
   (cd std && ./build.ss)
 }
 
+build_libgerbil () {
+  feedback_low "Building libgerbil"
+  PATH="${GERBIL_BASE}/bin:${PATH}"
+  GERBIL_HOME="${GERBIL_BASE}" #required for build
+  export PATH GERBIL_HOME
+  ./build/build-libgerbil.ss
+}
+
 build_lang () {
   feedback_low "Building gerbil languages"
   PATH="${GERBIL_BASE}/bin:${PATH}"
@@ -226,6 +234,7 @@ build_gerbil() {
   stage0           || die
   stage1 final     || die
   build_stdlib     || die
+  build_libgerbil  || die
   build_lang       || die
   build_r7rs_large || die
   build_tools      || die
@@ -252,6 +261,9 @@ else
        "stdlib")
          build_stdlib || die
          ;;
+       "libgerbil")
+         build_libgerbil || die
+         ;;
        "lang")
          build_lang || die
          ;;
@@ -273,7 +285,7 @@ else
        *)
          feedback_err "Unknown command."
          feedback_err \
-           "Correct usage: ./build.sh [gxi|stage0|stage1 [final]|stdlib|lang|r7rs-large|tools|tags]"
+           "Correct usage: ./build.sh [gxi|stage0|stage1 [final]|stdlib|libgerbil|lang|r7rs-large|tools|tags]"
          die
          ;;
   esac