random-state does not compile on Allegro Common Lisp

[rpgoldman]

The Error

Trying to compile random-state from master (put a clone in ~/quicklisp/local-projects/ gives me the following error:

; Fast loading
;    /Users/rpg/.cache/common-lisp/acl-11.0.pre-beta.99-macosx-64-bit_apple_silicon/Users/rpg/lisp/shop/jenkins/ext/random-state/implementation.fasl
Error: No next method for method
       #<STANDARD-METHOD %MAKE-GENERATOR (SYMBOL)> of generic function
       #<STANDARD-GENERIC-FUNCTION %MAKE-GENERATOR> with args (:SYSTEM)
  [condition type: PROGRAM-ERROR]

Restart actions (select using :continue):
  0: retry the load of /Users/rpg/.cache/common-lisp/acl-11.0.pre-beta.99-macosx-64-bit_apple_silicon/Users/rpg/lisp/shop/jenkins/ext/random-state/implementation.fasl
  1: skip loading /Users/rpg/.cache/common-lisp/acl-11.0.pre-beta.99-macosx-64-bit_apple_silicon/Users/rpg/lisp/shop/jenkins/ext/random-state/implementation.fasl
  2: Recompile implementation and try loading it again
  3: Retry
     loading FASL for #<CL-SOURCE-FILE "random-state" "implementation">.
  4: Continue, treating
     loading FASL for #<CL-SOURCE-FILE "random-state" "implementation">
     as having been successful.
  5: Retry ASDF operation.
  6: Retry ASDF operation after resetting the configuration.
  7: Give up on "random-state"
  8: Register local projects and try again.
  9: Return to Top Level (an "abort" restart).
 10: Abort entirely from this (lisp) process.

[changing package from "COMMON-LISP-USER" to "RANDOM-STATE"]
[1] ORG.SHIRAKUMO.RANDOM-STATE(3): :bt                                :bt
Evaluation stack:

(METHOD NO-NEXT-METHOD (STANDARD-GENERIC-FUNCTION STANDARD-METHOD)) <-
  (FLET #:ENSURE-GENERIC-FUNCTION-USING-CLASS--NULL EXCL::INITIAL-CHEAP-SELECT-FUNCTION) <- (METHOD %MAKE-GENERATOR (SYMBOL)) <-
  (:INTERNAL (:EFFECTIVE-METHOD 1 T ...) 0) <- MAKE-GENERATOR <-
  (:TOP-LEVEL-FORM "/Users/rpg/lisp/shop/jenkins/ext/random-state/implementation.lisp" 233) <- LOAD <- (:INTERNAL ASDF/LISP-BUILD:LOAD* 0) <-
  UIOP/UTILITY:CALL-WITH-MUFFLED-CONDITIONS <- ASDF/LISP-BUILD:CALL-WITH-MUFFLED-LOADER-CONDITIONS <- ASDF/LISP-BUILD:LOAD* <-
  ASDF/LISP-ACTION:PERFORM-LISP-LOAD-FASL <- (METHOD ASDF-ACTION:PERFORM (ASDF/LISP-ACTION:LOAD-OP ASDF/LISP-ACTION:CL-SOURCE-FILE)) <-
  (:INTERNAL (:EFFECTIVE-METHOD 2 NIL ...) 0) <- [... ASDF-ACTION:PERFORM ] <- (METHOD ASDF-ACTION:PERFORM-WITH-RESTARTS (T T)) <-
  (METHOD ASDF-ACTION:PERFORM-WITH-RESTARTS (ASDF/LISP-ACTION:LOAD-OP ASDF/LISP-ACTION:CL-SOURCE-FILE)) <-
  (METHOD ASDF-ACTION:PERFORM-WITH-RESTARTS :AROUND ...) <- (:INTERNAL (:EFFECTIVE-METHOD 2 NIL ...) 0) <-
  [... ASDF-ACTION:PERFORM-WITH-RESTARTS ] <- (METHOD ASDF/PLAN:PERFORM-PLAN (LIST)) <-
  (FLET (METHOD ASDF/PLAN:PERFORM-PLAN :AROUND ...) EXCL::CONTINUATION) <- (METHOD ASDF/PLAN:PERFORM-PLAN :AROUND ...) <-
  (:INTERNAL (:EFFECTIVE-METHOD 1 T ...) 0) <- (METHOD ASDF/PLAN:PERFORM-PLAN (T)) <-
  (FLET (METHOD ASDF/PLAN:PERFORM-PLAN :AROUND ...) EXCL::CONTINUATION) <- (METHOD ASDF/PLAN:PERFORM-PLAN :AROUND ...) <-
  (:INTERNAL (:EFFECTIVE-METHOD 1 T ...) 0) <- (METHOD ASDF/OPERATE:OPERATE (ASDF/OPERATION:OPERATION ASDF/COMPONENT:COMPONENT)) <-
  (:INTERNAL (:EFFECTIVE-METHOD 2 T ...) 0) <- (:INTERNAL (METHOD ASDF/OPERATE:OPERATE :AROUND ...) 0) <- ASDF/CACHE:CALL-WITH-ASDF-CACHE
  (METHOD ASDF/OPERATE:OPERATE :AROUND ...) <-  ...

Possible Cause

make-generator is invoked as a side-effect of the following line in implementation.lisp:

(setf (global-generator :system) *random-state*)

That causes invocation of (method %make-generator ((type symbol)), in generator.lisp, which first tries to find the symbol named "SYSTEM" in the random-state package. It does not find this, so it falls through to (call-next-method) which fails with the above error.

Honestly, I don't know why this works on SBCL, but it does. Interestingly, when I invoke (%make-generator :system) at the repl in SBCL, I get the same error, which suggests that there might be something different in how

(setf (global-generator :system) *random-state*)

works on Allegro and SBCL.

Note that you can test on Allegro (at least the hobbyist version) by using one of the Docker images that Eric Timmons has created for the various implementations, either directly or in GitHub actions.

Also note that I have been running this on Allegro for the M1, which is brand new, so it's possible that this is a bug in the lisp implementation.

[rpgoldman]

Could this be related to the use of load-time-value in the compiler macro for global-generator?

Also the entry for :system in *generators* is set on SBCL and not set on Allegro.

[Shinmera]

The compiler macro isn't used, this is a setf function being invoked here.

[Shinmera]

Or at least it should be. I don't know why Allegro is using the compiler macro.

[Shinmera]

Looks very much like a miscompilation to me? I see nothing problematic in the code.

[rpgoldman]

I don't know -- what seems to be wrong is that there is no entry for :system on Allegro, but there is on SBCL, but I can't figure out what is setting that, since calling (%make-generator :system) on SBCL errors for me, too. So it seems like something is setting that entry on SBCL without invoking %make-generator

[Shinmera]

Well exactly that line in implementation.lisp that it's choking on. (setf (global-generator :system) *random-state*) creates that entry with the setf function defined in generator.lisp

[rpgoldman]

Very weird, because when I do this at the ACL repl, I get the right result:

(setf (global-generator :system) *random-state*)

[Shinmera]

Yeah, so the ACL file-compiler on M1 is broken, expanding the compiler macro for some reason, instead of invoking the setf function.

[rpgoldman]

For now, can I just #-allegro the compiler macro? Will that break anything?

[Shinmera]

No, it just removes an optimisation around accessing it.

[rpgoldman]

I thought that was generally true about compiler macros, but wanted to check. Thanks!

I will report this to Franz.

[rpgoldman]

BTW, have you tested this on older Allegro? If I could report that it worked there, that would probably help them

[Shinmera]

I have not, no. I generally don't have the time to test on other implementations

[rpgoldman]

OK, when I get a chance, I can pass you a GitHub action script that will test on a bunch of implementations.

[fosskers]

I have just hit this as well, although the source seems to be different.

First of all, I needed to move the definition site of *generator* to package.lisp:

(in-package :random-state)
(defvar *generator* *random-state*)

Second, there's something odd going on with macro expansion. The compiler reports:

Attempt to take the value of the unbound variable
`RANDOM-STATE:RANDOM-INT'.

Complaining about something that has expanded to (coerce 0 random-int). The only place like that is:

(define-compiler-macro random-unit (&whole whole generator &optional (type ''single-float) &environment env)
  (if (constantp type env)
      `(scale-float (coerce (random-bytes ,generator (load-time-value (float-digits (coerce 0 ,type)))) ,type)
        (load-time-value (- (float-digits (coerce 0 ,type)))))
      whole))

which only seems to be called here:

(defun random-float (generator from to)
  (let ((from from)
        (to to))
    (when (< to from)
      (rotatef from to))
    (+ from (* (- to from) (random-unit generator (type-of from))))))

which likewise seems only called here:

(defun random (max &optional (generator *generator*))
  (macrolet ((gen (&rest types)
               `(etypecase max
                  ((integer 0)
                   (random-int generator 0 (1- max)))
                  ,@(loop for (type alias) in types
                          for zero = (coerce 0 type)
                          unless (and alias (subtypep type alias))
                            collect `((,type ,zero)
                                      (random-float generator ,zero max))))))
    (gen (short-float single-float)
         (single-float)
         (double-float)
         (long-float double-float))))

Otherwise, hard to say what's occurring.

By extension this is also affecting anything that depends on FSet.

[rpgoldman]

@fosskers -- on Allegro, as well? Or some other CL implementation.

I think Franz may have patched this issue -- incorrect use of compiler macro -- in their compiler (Allegro 11), but haven't had time to check.

[fosskers]

It's on Allegro 11 that I'm seeing the above issue.

[slburson]

@fosskers As a workaround for FSet, you can go into fset.asd and remove dependency :random-state from FSet, and file "testing" from module Code. You'll also need to go into Code/defs.lisp and remove the shadowing-import of random. You won't be able to build FSet/test anymore, but perhaps you can live with that.

[fosskers]

@slburson is the use of random-state necessary for the library itself, or only for testing? Is there some system reorganization that could fix this?

[Shinmera]

Personally I'd rather someone suggest a way to #+allegro fix random-state :)

[slburson]

@fosskers It's just for testing. I've now pushed a reversion to FSet that switches it back to MT19937, which it used for years.

@Shinmera Looking at the 1/17 comment above, I think something is badly broken in Allegro's handling of compiler macros. random-int is a function name; it's not passed as a type argument to random-unit. How it could have wound up in that coerce form is hard to fathom. I think your only hope is just to #-allegro all your compiler macros.

[rpgoldman]

There was a bug in Allegro's handling of compiler macros. A fully-patched Allegro 11.0 should, I think, not encounter this error.

@fosskers Does this error persist after (sys:update-allegro)?

[rpgoldman]

I just loaded random-state in a patched Allegro 11.0 and it works fine now.

Unless someone else can replicate this bug in a fully-patched Allegro 11, I suggest we close it.

[fosskers]

I will test this as soon as I get word back from Franz about Allegro being busted on Arch.

[fosskers]

Turns out it's due to the new glibc, as reported elsewhere: JuliaLang/julia#57250

I'm able to compile both random-state and fset now, as well as run my downstream tests.

[slburson]

@fosskers What did you have to do, exactly? Did you just get another update from Franz, or did you do this execstack -c thing, or what?

[fosskers]

I ran execstack -c on the problematic .so shipped with Allegro, then ran update.sh twice (with another execstack in between) to finally get a fully patched Allegro that runs. The Franz guys are fully aware of the issue now though.