Pure Gerbil Bootstrap (#880)

Now that we have the Pure Gerbil Runtime, we can complete the bootstrap
loop.

RIP gx-gambc; you served us well.

.github/workflows/ci.yml

@@ -16,8 +16,7 @@ jobs:
         run: ./configure --prefix=/opt/gerbil ${{ matrix.gambit-version }} ${{ matrix.shared-mode }}
       - name: Create Gerbil version file
         run: |
-             echo "(define (gerbil-version-string) \"git-$(git rev-parse --short=8 HEAD)\")" > src/gerbil/boot/runtime/gx-version.scm
-             echo "(define (gerbil-version-string) \"git-$(git rev-parse --short=8 HEAD)\")" > src/gerbil/runtime/version.ss
+              echo "(def (gerbil-version-string) \"git-$(git rev-parse --short=8 HEAD)\")" > src/gerbil/runtime/version.ss
       - name: Prepare Build
         run: |
            . ./build-env.sh

.gitignore

@@ -13,9 +13,9 @@ src/TAGS
 .build*
 doc/node_modules
 doc/.vuepress/dist
-src/gerbil/boot/runtime/gx-version.scm
 src/gerbil/runtime/version.ss
 src/gerbil/boot-gxi
+src/bootstrap/static
 
 build-deps
 build-env.sh

build.sh

@@ -14,4 +14,4 @@ else
     die
 fi
 
-(cd src && ./build.sh) || die
+(cd src && ./build.sh $@) || die

doc/.vuepress/config.js

@@ -14,7 +14,7 @@ module.exports = {
           '/guide/': [
               {
                   title: 'Guide',
-                  children: ['', 'intro', 'getting-started', 'emacs', 'shell', 'env-vars', 'package-manager', 'docker', 'nix', 'r7rs', 'macos']
+                  children: ['', 'intro', 'getting-started', 'ffi', 'emacs', 'shell', 'env-vars', 'package-manager', 'docker', 'nix', 'r7rs', 'macos']
               }
           ],
           '/tutorials/': [
@@ -197,7 +197,6 @@ module.exports = {
                       'test',
                       'debug',
                       'profiler',
-                      'ffi',
                       'bootstrap'
                   ]
               },

doc/reference/dev/bootstrap.md

@@ -1,32 +1,163 @@
-# Updating Bootstrap
+# The Gerbil Bootstrap
 
-If you have made changes to the core prelude, expander, or the
-compiler, sometimes you may also need to update the compiled bootstrap
-modules (e.g. because of a bug fix).
+Gerbil is fully self-hosted, 100% written in itself. So how does it
+all fit together?
+
+## The Chain of Trust
+
+The key premise of Gerbil is simple: it is a meta-language, a
+meta-dialect of Scheme that bootstraps from precompiled sources using
+Gambit.
+
+This has implications for the chain of trust regarding the security of
+your software:
+- There is _no precompiled binary involved_, nor will there ever be
+  one. The bootstrap is purely source based, which means you can
+  actually read and audit the bootstrap sources. It is not pretty, but
+  it is readable; so if you feel so inclined I recommend taking a look
+  at `GERBIL_SRCDIR/src/bootstrap`.
+- Just like Gerbil bootstraps from precompiled Gambit code, Gambit
+  bootstraps from precompiled C code. This is also auditable, albeit
+  not an easy read.
+- The bootstrap chain is anchored on the C compiler. Ultimately, If
+  you trust your C compiler, then you can _verifiably_ trust the
+  Gerbil bootstrap.
+
+For the Gerbil core team, where we all use GCC, this can be
+summarized in a quotable one liner:
+
+> In GNU we trust; everyone else pays cash.
+
+## The Long and Arduous History of Bootstrap
+
+The first version of the Gerbil, let's call that the proto-Gerbil, was
+bootstrapped by vyzo a long time ago using a hand-written unhygienic
+interpreter for the core language.  Once that was done, vyzo wrote the
+expander and the first version of the compiler, then the expander
+expanded the compiler and the compiler compiled the expander and
+itself... and Gerbil was born.
+
+Initially, the runtime was written in Gambit with a set of macros;
+that was called `gx-gambc`.  In the v0.18 release cycle, where Gerbil
+became fully self hosted, all the traces have disappeared from the
+source tree, as they are dead code. They still exist in the
+repo's commit history if you want to do some historical research and
+peek into the deep past to understand the evolution of Gerbil.
+
+
+## How Gerbil Builds Itself
+
+The build process can be summarized in the following steps:
+1. `configure` configures the system and creates `build-env.sh`.
+2. `make` invokes the [top level build script](https://github.com/mighty-gerbils/gerbil/blob/master/build.sh)
+3. `build.sh` sources `build-env.sh` and dispatches to the [worker build script](https://github.com/mighty-gerbils/gerbil/blob/master/src/build.sh).
+4. The worker `build.sh` sets up the build environment and proceeds in stages:
+   1. the build directory structure is prepared
+   2. Gambit is bootstrapped and built.
+   3. the `boot-gxi` shim is compiled using `gcc`.
+   4. the Gerbil bootstrap is compiled to object code using `gsc` (stage 0).
+   5. the Gerbil core system and universal binary is compiled using the bootstrap compiler with `boot-gxi` (stage 1).
+   6. the newly compiled `gerbil` binary compiles the rest of the system.
+
+
+## Practical Matters
+
+### Recompiling the Bootstrap
+If you have made changes to the core runtime, prelude, expander, or
+the compiler itself, then you may also need to update the precompiled
+bootstrap modules (e.g. because of a bug fix).
 
 This can be accomplished with the following incantations in `$GERBIL_SRCDIR/src`.
 
-To compile the core prelude bootstrap:
+- To compile the bootstrap runtime:
+```
+gxc -d bootstrap -s -S -O gerbil/runtime/{gambit,system,util,loader,control,mop,error,syntax,eval,repl,init}.ss gerbil/runtime.ss
+```
+
+- To compile the bootstrap core prelude:
 ```
 gxc -d bootstrap -s -S -O -no-ssxi gerbil/prelude/core.ss
 ```
 
-To compile the gambit prelude bootstrap:
+- To compile the bootstrap gambit prelude:
 ```
 gxc -d bootstrap -s -S gerbil/prelude/gambit/*.ss gerbil/prelude/gambit.ss
 ```
 
-To compile the expander bootstrap:
+- To compile the bootstrap expander:
 ```
 gxc -d bootstrap -s -S -O gerbil/expander/{common,stx,core,top,module,compile,root,stxcase}.ss gerbil/expander.ss
 ```
 
-To compile the compiler bootstrap:
+- To compile the bootstrap compiler:
 ```
 gxc  -d bootstrap -s -S -O gerbil/compiler/{base,compile,optimize-base,optimize-xform,optimize-top,optimize-spec,optimize-ann,optimize-call,optimize,driver,ssxi}.ss gerbil/compiler.ss
 ```
 
-Finally, if you've made changes to it, you should also copy the core.ssxi.ss optimizer prelude:
+- Finally, if you've made changes to it, you should also copy the core.ssxi.ss optimizer prelude:
 ```
 cp gerbil/prelude/core.ssxi.ss bootstrap/gerbil
 ```
+
+### Debugging
+
+If you have been making changes in the core system and building a new
+bootstrap, you might have some bugs and need to debug. The gerbil
+build script is modular, which allows you to build certain stages at
+will and supports serveral commands:
+- `prepare` prepares the build environment.
+- `gambit` bootstraps and builds Gambit.
+- `boot-gxi` builds and bootstrap gxi shim.
+- `stage0` builds the Gerbil bootstrap.
+- `stage1` builds the Gerbil core modules and the `gerbil` universal
+  binary (aka Bach, affectionally named after the great composer of
+  timeless music).
+- `stdlib` builds the Gerbil standard library.
+- `libgerbil` builds the Gerbil (shared) library; this includes the
+  Gerbil runtime, prelude, expander and compiler, together with the
+  entiry of stdlib.
+- `lang` builds the bundled alternative language preludes.
+- `r7rs-large` builds the R7RS large libraries included in Gerbil.
+- `srfi` builds the top level srfi package shims.
+- `tools` builds the gerbil tools.
+- `tags` builds a TAGS file for the Gerbil sources so that you can
+  easily navigate code in emacs.
+- `env` applies the arguments in the build environment.
+
+So if you have made changes and want to rebuild gerbil, you don't have
+to redo everything from scratch with `make`; you can simply build the
+stage you want, and once you are satisfied you can move to the next
+stage or push your branch so that CI does the job for you.
+
+If you have introduced a bug and `gerbil` crashes, do not despair. You
+can simply run with `gdb` and get the exact location of the crash to
+pinpoint the bug.
+
+Here is an example from debugging a real crash during development:
+```
+$ ./build.sh stdlib
+[*] Building gerbil stdlib
+Segmentation fault (core dumped)
+
+$ ./build.sh env gdb gerbil
+...
+For help, type "help".
+Type "apropos word" to search for commands related to "word"...
+Reading symbols from gerbil...
+(gdb) r ./std/build.ss
+Starting program: /home/vyzo/gerbil/build/bin/gerbil ./std/build.ss
+[Thread debugging using libthread_db enabled]
+Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
+
+Program received signal SIGSEGV, Segmentation fault.
+___H_gerbil____runtime____gambit (___ps=0x7ffff7eda000 <___gstate0>) at /home/vyzo/gerbil/build/lib/static/gerbil__runtime__gambit.scm:4
+4	  (define ##max-char (let () (declare (not safe)) (##max-char-code))))
+```
+
+Once you have successfully debugged and finished the build, you should run the tests:
+```
+$ ./build.sh env gerbil test ./...
+...
+```
+
+And that's it! Happy Hacking.

src/bootstrap/gerbil/compiler/base.ssi

@@ -5,7 +5,7 @@ namespace: gxc
 (%#begin (%#import :gerbil/expander (in: :gerbil/core <syntax-case>))
          (%#export #t)
          (%#begin (%#begin-syntax
-                   (%#call (%#ref _gx#load-module)
+                   (%#call (%#ref load-module)
                            (%#quote "gerbil/compiler/base__1")))
                   (%#define-syntax ast-case |gxc[:0:]#ast-case|))
          (%#define-syntax ast-rules |gxc[:0:]#ast-rules|)
@@ -62,4 +62,4 @@ namespace: gxc
           module-path-reserved-chars
           gxc#module-path-reserved-chars)
          (%#define-runtime module-id->path-string gxc#module-id->path-string))
-(%#call (%#ref _gx#load-module) (%#quote "gerbil/compiler/base__0"))
+(%#call (%#ref load-module) (%#quote "gerbil/compiler/base__0"))

src/bootstrap/gerbil/compiler/base__0.scm

@@ -1,6 +1,6 @@
 (declare (block) (standard-bindings) (extended-bindings))
 (begin
-  (define gerbil/compiler/base::timestamp 1695117617)
+  (define gerbil/compiler/base::timestamp 1695199291)
   (begin
     (define gxc#current-compile-symbol-table (make-parameter '#f))
     (define gxc#current-compile-runtime-sections (make-parameter '#f))

src/bootstrap/gerbil/compiler/compile.ssi

@@ -12,13 +12,13 @@ namespace: gxc
            (0 u64vector? 0 u64vector?)
            (0 u32vector? 0 u32vector?)
            (0 u16vector? 0 u16vector?)
-           (0 s64vector? 0 s64vector?)
+           (0 f64vector? 0 f64vector?)
+           (0 s16vector? 0 s16vector?)
            (0 s8vector? 0 s8vector?)
            (0 s32vector? 0 s32vector?)
-           (0 f64vector? 0 f64vector?)
+           (0 s64vector? 0 s64vector?)
            (0 f32vector? 0 f32vector?)
-           (0 u8vector? 0 u8vector?)
-           (0 s16vector? 0 s16vector?)))
+           (0 u8vector? 0 u8vector?)))
          (%#export #t)
          (%#define-runtime current-compile-methods gxc#current-compile-methods)
          (%#define-runtime current-compile-lift gxc#current-compile-lift)
@@ -34,7 +34,7 @@ namespace: gxc
           gxc#make-bound-identifier-table)
          (%#define-runtime compile-e gxc#compile-e)
          (%#begin (%#begin-syntax
-                   (%#call (%#ref _gx#load-module)
+                   (%#call (%#ref load-module)
                            (%#quote "gerbil/compiler/compile__1")))
                   (%#define-syntax
                    defcompile-method
@@ -331,4 +331,4 @@ namespace: gxc
           gxc#count-values-let-values%)
          (%#define-runtime count-values-call% gxc#count-values-call%)
          (%#define-runtime count-values-if% gxc#count-values-if%))
-(%#call (%#ref _gx#load-module) (%#quote "gerbil/compiler/compile__0"))
+(%#call (%#ref load-module) (%#quote "gerbil/compiler/compile__0"))

src/bootstrap/gerbil/compiler/compile__0.scm

@@ -1,6 +1,6 @@
 (declare (block) (standard-bindings) (extended-bindings))
 (begin
-  (define gerbil/compiler/compile::timestamp 1695117617)
+  (define gerbil/compiler/compile::timestamp 1695199291)
   (begin
     (define gxc#current-compile-methods (make-parameter '#f))
     (define gxc#current-compile-lift (make-parameter '#f))

src/bootstrap/gerbil/compiler/driver.ssi

@@ -12,13 +12,13 @@ namespace: gxc
           (spec:
            (:gerbil/gambit/ports)
            (0 open-process 0 open-process)
-           (0 close-port 0 close-port)
-           (0 process-status 0 process-status))
+           (0 process-status 0 process-status)
+           (0 close-port 0 close-port))
           (spec:
            (:gerbil/gambit/os)
            (0 current-time 0 current-time)
-           (0 file-info-size 0 file-info-size)
            (0 time->seconds 0 time->seconds)
+           (0 file-info-size 0 file-info-size)
            (0 file-info 0 file-info))
           (spec:
            (:gerbil/gambit/threads)
@@ -32,7 +32,7 @@ namespace: gxc
          (%#define-runtime default-gerbil-ar gxc#default-gerbil-ar)
          (%#define-runtime +driver-mutex+ gxc#+driver-mutex+)
          (%#begin (%#begin-syntax
-                   (%#call (%#ref _gx#load-module)
+                   (%#call (%#ref load-module)
                            (%#quote "gerbil/compiler/driver__1")))
                   (%#define-syntax
                    with-driver-mutex
@@ -49,7 +49,6 @@ namespace: gxc
          (%#define-runtime gerbil-gcc gxc#gerbil-gcc)
          (%#define-runtime +gerbil-ar+ gxc#+gerbil-ar+)
          (%#define-runtime gerbil-ar gxc#gerbil-ar)
-         (%#define-runtime gsc-runtime-args gxc#gsc-runtime-args)
          (%#define-runtime gerbil-runtime-modules gxc#gerbil-runtime-modules)
          (%#define-runtime delete-directory* gxc#delete-directory*)
          (%#begin (%#define-runtime compile-module__% gxc#compile-module__%)
@@ -112,4 +111,4 @@ namespace: gxc
          (%#define-runtime compile-exe-output-file gxc#compile-exe-output-file)
          (%#define-runtime static-module-name gxc#static-module-name)
          (%#define-runtime invoke gxc#invoke))
-(%#call (%#ref _gx#load-module) (%#quote "gerbil/compiler/driver__0"))
+(%#call (%#ref load-module) (%#quote "gerbil/compiler/driver__0"))