Welcome to the future, Part I (#776)

* libgerbil as a shared library

build and use libgerbil as an .so when --enable-shared

* update documentation about libgergil.so

* set rpath for dynhamically linked executables

* The Gerbil Universal Binary, aka BACH

This takes us halfway to deep integration; we now have a universal
gerbil binary, which links and initializes the expander/compiler and
it is capable of acting as any gerbil tool with a softlink.

We only depend on gsi in bootstreap (boot-gxi), which gets as far as
building bach. After that, the snake eats its tail.

* fix variable name for consistency

* fix CI build

* some cosmetic improvements

and better handling of --lang argument in gxi

* rename gxlink to libgerbil-link

* RIP dynamic exe stubs

- The restriction on (formerly known as) static binaries is lifted;
  the expander and compiler are now included in libgerbil.
- With that restriction lifted, dynamic executable stubs are no longer
  needed and are unceremoniously removed. We just have executables now.

* compile-file is now compile-module

* fix crypto-test

* static all the time

* fully expand user static lib dir

* add a check in crypto-test

* env variables for external toolchain

* parallelize libgerbil build

* kill build-deps files

* and bury them

* regenerate bootstrap

This is the last bootstrap before Deep Gerbil!

new bootstrap files

* update bootstrap build instructions

* update boot build scripts for the new bootstrap

* std/misc/wg: wrap exception stack trace around workers

* update bach for new bootstrap

.github/workflows/ci.yml

@@ -49,8 +49,8 @@ jobs:
           gsc -v
       - name: Setup Gerbil build
         run: echo "(define (gerbil-version-string) \"git-$(git rev-parse --short=8 HEAD)\")" > src/gerbil/runtime/gx-version.scm
-      - name: Build Gerbil gxi
-        run: ./src/build.sh gxi
+      - name: Build Gerbil boot gxi
+        run: ./src/build.sh boot-gxi
       - name: Build Gerbil stage0
         run: ./src/build.sh stage0
       - name: Build Gerbil stage1 final

.gitignore

@@ -13,9 +13,6 @@ src/TAGS
 doc/node_modules
 doc/.vuepress/dist
 src/gerbil/runtime/gx-version.scm
-src/gerbil/gxi
+src/gerbil/boot-gxi
 
-src/tutorial/lang/build-deps
-src/tutorial/proxy/build-deps
-src/tutorial/proxy/socks-proxy
-src/tutorial/proxy/tcp-proxy
+build-deps

doc/guide/bootstrap.md

@@ -8,15 +8,23 @@ To compile the core prelude bootstrap:
 ```
 gxc -d bootstrap -s -S -O -no-ssxi gerbil/prelude/core.ss
 ```
+
 To compile the gambit prelude bootstrap:
 ```
 gxc -d bootstrap -s -S gerbil/prelude/gambit/*.ss gerbil/prelude/gambit.ss
 ```
+
 To compile the expander bootstrap:
 ```
 gxc -d bootstrap -s -S -O gerbil/expander/{common,stx,core,top,module,compile,root,stxcase}.ss gerbil/expander.ss
 ```
+
 To compile the compiler bootstrap:
 ```
-gxc  -d bootstrap -s -S -O gerbil/compiler/{base,compile,optimize-base,optimize-xform,optimize-top,optimize-ann,optimize-call,optimize,driver,ssxi}.ss gerbil/compiler.ss
+gxc  -d bootstrap -s -S -O gerbil/compiler/{base,compile,optimize-base,optimize-xform,optimize-top,optimize-spec,optimize-ann,optimize-call,optimize,driver,ssxi}.ss gerbil/compiler.ss
+```
+
+Finally, if you've made changes to it, you should also copy the core.ssxi.ss optimizer prelude:
+```
+cp gerbil/prelude/core.ssxi.ss bootstrap/gerbil
 ```

doc/guide/getting-started.md

@@ -134,10 +134,13 @@ alternatively, you could symlink the installed `gxi` and `gxc` binaries
 into a directory already in your `$PATH`.
 You may also define `GERBIL_HOME`, or leave it undefined (or unexported)
 and let Gerbil autodetect where it was installed.
+
 I (vyzo) have the following in my `.bashrc`:
 ```
 export GERBIL_HOME=$HOME/gerbil
 add_path $GERBIL_HOME/bin
+add_ldpath $GERBIL_HOME/lib
+add_path $HOME/.gerbil/bin
 ```
 
 ## Write some code

doc/guide/intro.md

@@ -806,21 +806,23 @@ hello world
 
 The difference between the 3 executable compilation modes can be summarized as follows:
 - By default, a statically linked executable is generated, linking to the precompiled
-  gerbil standard library. Note that the executable may some have dynamic library
-  dependencies from stdlib foreign code , and also links to `libgambit`.
-  If you have configured your   gambit with `--enable-shared`, then this will be
-  a dynamic library dependency.
+  gerbil standard library. If the system was configured with `--enable-shared`, then this
+  will be a shared library; otherwise it will be a static library archive.
+  Note that the executable may some have additionl dynamic library
+  dependencies from stdlib foreign code , and also links to `libgambit` which will be
+  a shared library when the system is configured with `--enable-shared`.
 - When `-full-program-optimization` is passed to `gxc`, then the compiler will perform
   full program optimization with all gerbil library dependencies. This will result
-  both in smaller executable size and better performance, albeit at the cost of
-  increased compilation time; this can be minutes for complex programs, while
+  both in better performance, albeit at the cost of increased compilation time;
+  this can be minutes for complex programs, while
   separately linked executables compile in a second. Furthermore, because
   dependencies are compiled in together, you can apply declarations like `(not safe)`
   to the whole program using the `-prelude` directive. This can result
   in potentially significant performance gains at the expense of safety.
 - When `-dynamic` is passed to `gxc`, then a dynamic executable stub will be generated,
   which will depend on the Gerbil runtime environment being present at execution time.
-  Dynamic executables do have some advantages over static executables however:
+  Dynamic executables are very useful for development and do have some advantages over
+  static executables:
   - they compile instantly and are tiny
   - they can use the expander and the compiler; note that this restriction will be
     lifted from static executables in a future release.

src/bootstrap/gerbil/compiler/base.ssi

@@ -42,6 +42,7 @@ namespace: gxc
          (%#define-runtime
           current-compile-timestamp
           gxc#current-compile-timestamp)
+         (%#define-runtime current-compile-decls gxc#current-compile-decls)
          (%#define-runtime symbol-table::t gxc#symbol-table::t)
          (%#define-runtime symbol-table? gxc#symbol-table?)
          (%#define-runtime make-symbol-table gxc#make-symbol-table)

src/bootstrap/gerbil/compiler/base__0.scm

@@ -1,73 +1,86 @@
 (declare (block) (standard-bindings) (extended-bindings))
 (begin
-  (define gxc#current-compile-symbol-table (make-parameter '#f))
-  (define gxc#current-compile-runtime-sections (make-parameter '#f))
-  (define gxc#current-compile-runtime-names (make-parameter '#f))
-  (define gxc#current-compile-output-dir (make-parameter '#f))
-  (define gxc#current-compile-invoke-gsc (make-parameter '#f))
-  (define gxc#current-compile-gsc-options (make-parameter '#f))
-  (define gxc#current-compile-keep-scm (make-parameter '#f))
-  (define gxc#current-compile-verbose (make-parameter '#f))
-  (define gxc#current-compile-optimize (make-parameter '#f))
-  (define gxc#current-compile-debug (make-parameter '#f))
-  (define gxc#current-compile-generate-ssxi (make-parameter '#f))
-  (define gxc#current-compile-static (make-parameter '#f))
-  (define gxc#current-compile-timestamp (make-parameter '#f))
-  (define gxc#symbol-table::t
-    (make-struct-type
-     'gxc#symbol-table::t
-     '#f
-     '2
-     'symbol-table
-     '()
-     ':init!
-     '(gensyms bindings)))
-  (define gxc#symbol-table? (make-struct-predicate gxc#symbol-table::t))
-  (define gxc#make-symbol-table
-    (lambda _$args876_
-      (apply make-struct-instance gxc#symbol-table::t _$args876_)))
-  (define gxc#symbol-table-gensyms
-    (make-struct-field-accessor gxc#symbol-table::t '0))
-  (define gxc#symbol-table-bindings
-    (make-struct-field-accessor gxc#symbol-table::t '1))
-  (define gxc#symbol-table-gensyms-set!
-    (make-struct-field-mutator gxc#symbol-table::t '0))
-  (define gxc#symbol-table-bindings-set!
-    (make-struct-field-mutator gxc#symbol-table::t '1))
-  (define gxc#symbol-table:::init!
-    (lambda (_self874_)
-      (if (##fx< '2 (##vector-length _self874_))
-          (begin
-            (##vector-set! _self874_ '1 (make-table 'test: eq?))
-            (##vector-set! _self874_ '2 (make-table 'test: eq?)))
-          (error '"struct-instance-init!: too many arguments for struct"
-                 _self874_))))
-  (bind-method! gxc#symbol-table::t ':init! gxc#symbol-table:::init! '#f)
-  (define gxc#raise-compile-error
-    (lambda (_message748_ _stx749_ . _details750_)
-      (apply gx#raise-syntax-error
-             'compile
-             _message748_
-             _stx749_
-             _details750_)))
-  (define gxc#verbose
-    (lambda _args746_
-      (if (gxc#current-compile-verbose) (apply displayln _args746_) '#!void)))
-  (define gxc#module-path-reserved-chars '":#<>&!?*;()[]{}|'`\"\\")
-  (define gxc#module-id->path-string
-    (lambda (_id728_)
-      (let* ((_str730_ (if (symbol? _id728_) (symbol->string _id728_) _id728_))
-             (_len732_ (string-length _str730_))
-             (_res734_ (make-string _len732_)))
-        (let _lp737_ ((_i739_ '0))
-          (if (fx< _i739_ _len732_)
-              (let* ((_char741_ (string-ref _str730_ _i739_))
-                     (_xchar743_
-                      (if (string-index
-                           gxc#module-path-reserved-chars
-                           _char741_)
-                          '#\_
-                          _char741_)))
-                (string-set! _res734_ _i739_ _xchar743_)
-                (_lp737_ (fx+ _i739_ '1)))
-              _res734_))))))
+  (define gerbil/compiler/base::timestamp 1693679900)
+  (begin
+    (define gxc#current-compile-symbol-table (make-parameter '#f))
+    (define gxc#current-compile-runtime-sections (make-parameter '#f))
+    (define gxc#current-compile-runtime-names (make-parameter '#f))
+    (define gxc#current-compile-output-dir (make-parameter '#f))
+    (define gxc#current-compile-invoke-gsc (make-parameter '#f))
+    (define gxc#current-compile-gsc-options (make-parameter '#f))
+    (define gxc#current-compile-keep-scm (make-parameter '#f))
+    (define gxc#current-compile-verbose (make-parameter '#f))
+    (define gxc#current-compile-optimize (make-parameter '#f))
+    (define gxc#current-compile-debug (make-parameter '#f))
+    (define gxc#current-compile-generate-ssxi (make-parameter '#f))
+    (define gxc#current-compile-static (make-parameter '#f))
+    (define gxc#current-compile-timestamp (make-parameter '#f))
+    (define gxc#current-compile-decls (make-parameter '#f))
+    (define gxc#symbol-table::t
+      (make-struct-type
+       'gxc#symbol-table::t
+       '#f
+       '2
+       'symbol-table
+       '()
+       ':init!
+       '(gensyms bindings)))
+    (define gxc#symbol-table? (make-struct-predicate gxc#symbol-table::t))
+    (define gxc#make-symbol-table
+      (lambda _$args534_
+        (apply make-struct-instance gxc#symbol-table::t _$args534_)))
+    (define gxc#symbol-table-gensyms
+      (make-struct-field-accessor gxc#symbol-table::t '0))
+    (define gxc#symbol-table-bindings
+      (make-struct-field-accessor gxc#symbol-table::t '1))
+    (define gxc#symbol-table-gensyms-set!
+      (make-struct-field-mutator gxc#symbol-table::t '0))
+    (define gxc#symbol-table-bindings-set!
+      (make-struct-field-mutator gxc#symbol-table::t '1))
+    (define gxc#symbol-table:::init!
+      (lambda (_self532_)
+        (if (let ((__tmp4770
+                   (let () (declare (not safe)) (##vector-length _self532_))))
+              (declare (not safe))
+              (##fx< '2 __tmp4770))
+            (begin
+              (let ((__tmp4771 (make-table 'test: eq?)))
+                (declare (not safe))
+                (##vector-set! _self532_ '1 __tmp4771))
+              (let ((__tmp4772 (make-table 'test: eq?)))
+                (declare (not safe))
+                (##vector-set! _self532_ '2 __tmp4772)))
+            (error '"struct-instance-init!: too many arguments for struct"
+                   _self532_))))
+    (bind-method! gxc#symbol-table::t ':init! gxc#symbol-table:::init! '#f)
+    (define gxc#raise-compile-error
+      (lambda (_message406_ _stx407_ . _details408_)
+        (apply gx#raise-syntax-error
+               'compile
+               _message406_
+               _stx407_
+               _details408_)))
+    (define gxc#verbose
+      (lambda _args404_
+        (if (gxc#current-compile-verbose)
+            (apply displayln _args404_)
+            '#!void)))
+    (define gxc#module-path-reserved-chars '":#<>&!?*;()[]{}|'`\"\\")
+    (define gxc#module-id->path-string
+      (lambda (_id386_)
+        (let* ((_str388_
+                (if (symbol? _id386_) (symbol->string _id386_) _id386_))
+               (_len390_ (string-length _str388_))
+               (_res392_ (make-string _len390_)))
+          (let _lp395_ ((_i397_ '0))
+            (if (fx< _i397_ _len390_)
+                (let* ((_char399_ (string-ref _str388_ _i397_))
+                       (_xchar401_
+                        (if (string-index
+                             gxc#module-path-reserved-chars
+                             _char399_)
+                            '#\_
+                            _char399_)))
+                  (string-set! _res392_ _i397_ _xchar401_)
+                  (_lp395_ (fx+ _i397_ '1)))
+                _res392_)))))))