Trouble locating libncurses on MacOS with reline + jruby + fiddle FFI

[headius]

Hello!

We recently started pulling the latest stdlib gems into JRuby for Ruby 3.4 support and ran into an issue with reline loading libncurses via fiddle:

$ jruby -e 'require "reline"'
LoadError: Could not open library 'libncursesw.dylib' : dlopen(libncursesw.dylib, 0x0009): tried: 'libncursesw.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OSlibncursesw.dylib' (no such file), '/Library/Java/JavaVirtualMachines/temurin-17.jdk/Contents/Home/bin/./libncursesw.dylib' (no such file), '/Library/Java/JavaVirtualMachines/temurin-17.jdk/Contents/Home/bin/../lib/libncursesw.dylib' (no such file), '/usr/lib/libncursesw.dylib' (no such file, not in dyld cache), 'libncursesw.dylib' (no such file), '/usr/lib/libncursesw.dylib' (no such file, not in dyld cache)
        open at org/jruby/ext/ffi/jffi/DynamicLibrary.java:100
  initialize at /Users/headius/work/jruby/lib/ruby/gems/shared/gems/fiddle-1.1.3/lib/fiddle/ffi_backend.rb:478
         new at org/jruby/RubyClass.java:921
   curses_dl at /Users/headius/work/jruby/lib/ruby/stdlib/reline/terminfo.rb:41
        each at org/jruby/RubyArray.java:1954
   curses_dl at /Users/headius/work/jruby/lib/ruby/stdlib/reline/terminfo.rb:40
      <main> at /Users/headius/work/jruby/lib/ruby/stdlib/reline/terminfo.rb:152
     require at org/jruby/RubyKernel.java:1186
     require at /Users/headius/work/jruby/lib/ruby/stdlib/rubygems/core_ext/kernel_require.rb:136
      <main> at /Users/headius/work/jruby/lib/ruby/stdlib/reline.rb:9
     require at org/jruby/RubyKernel.java:1186
     require at /Users/headius/work/jruby/lib/ruby/stdlib/rubygems/core_ext/kernel_require.rb:136
      <main> at -e:1

Several locations are searched here but the file is not found. A few possible issues:

• A small patch provided below was necessary to get reline to look for the proper .dylib name rather than .so. RUBY_PLATFORM is java and RbConfig should be used to get the real host platform.
• I do not have libncurses in /usr/lib or /usr/local/lib but I am not sure if there are other places on MacOS I should be looking.
• I do have libncurses.dylib installed via Homebrew.
• If reline depends on libncurses unconditionally (as here, can't even load reline without libncurses present), perhaps some message on install is needed?
• This could be a bug in the FFI-based fiddle we are using on JRuby (and TruffleRuby). cc @kou @eregon

We are keen to figure this issue out so we can proceed with running the latest gems, stdlib, and tests in JRuby's 3.4-compatible branch.

Patch for reline to properly select a set of libncurses file names:

diff --git a/reline/terminfo.rb b/reline/terminfo.rb
--- a/reline/terminfo.rb	
+++ b/reline/terminfo.rb	(date 1729186018729)
@@ -20,7 +20,7 @@
   class TerminfoError < StandardError; end
 
   def self.curses_dl_files
-    case RUBY_PLATFORM
+    case RbConfig::CONFIG['host_os']
     when /mingw/, /mswin/
       # aren't supported
       []

[eregon]

Re RUBY_PLATFORM, it would be nice if JRuby would be compatible there, and be the native platform, i.e. like the [...] part in RUBY_DESCRIPTION: jruby/jruby#6152
Because it sounds like this workaround is and will be needed in many places.
(and on top of that JRuby docs suggest/ed to use RbConfig::CONFIG['host_os'] but apparently that's the wrong one, and should be target_os, I recall some discussion about this on the FFI & Ruby tracker).

Re not finding the library, I suspect that's because JRuby uses a rather old FFI which doesn't have this fix to search for Homebrew on darwin-arm64: ffi/ffi#968
Or that your Homebrew is in a non-standard location maybe.

[headius]

it would be nice if JRuby would be compatible there

JRuby has used "java" for RUBY_PLATFORM for almost twenty years and there are hundreds of gems out there that expect it to remain that way, as I described in jruby/jruby#6152.

We originally were forced to use "java" here because RubyGems could not support JRuby-specific extension gems any other way. I'm not sure when RubyGems switched to RbConfig::CONFIG['arch'] but in any case the cat was out of the bag long before the other Ruby implementations even existed. Claiming "java" as our platform also made sense for many other reasons, and it has long been JRuby's goal that applications work the same on Windows as on Unix.

I suspect that's because JRuby uses a rather old FFI

JRuby does not use a "rather old FFI". JRuby sources all FFI Ruby code from the gem, currently at version 1.16.3. That fix appears to have been released in 1.16.0.

That should be updated to 1.17.0 but the fix you describe should already be shipping with JRuby.

[headius]

Same result with FFI 1.17.0:

$ jruby -e 'require "ffi"; puts $".grep(/ffi.rb/); require "reline"'
/Users/headius/work/jruby/lib/ruby/gems/shared/gems/ffi-1.17.0-java/lib/ffi/ffi.rb
/Users/headius/work/jruby/lib/ruby/gems/shared/gems/ffi-1.17.0-java/lib/ffi.rb
LoadError: Could not open library 'libncursesw.dylib' : dlopen(libncursesw.dylib, 0x0009): tried: 'libncursesw.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OSlibncursesw.dylib' (no such file), '/Library/Java/JavaVirtualMachines/temurin-17.jdk/Contents/Home/bin/./libncursesw.dylib' (no such file), '/Library/Java/JavaVirtualMachines/temurin-17.jdk/Contents/Home/bin/../lib/libncursesw.dylib' (no such file), '/usr/lib/libncursesw.dylib' (no such file, not in dyld cache), 'libncursesw.dylib' (no such file), '/usr/lib/libncursesw.dylib' (no such file, not in dyld cache)
        open at org/jruby/ext/ffi/jffi/DynamicLibrary.java:100
  initialize at /Users/headius/work/jruby/lib/ruby/gems/shared/gems/fiddle-1.1.3/lib/fiddle/ffi_backend.rb:478
         new at org/jruby/RubyClass.java:921
   curses_dl at /Users/headius/work/jruby/lib/ruby/stdlib/reline/terminfo.rb:41
        each at org/jruby/RubyArray.java:1954
   curses_dl at /Users/headius/work/jruby/lib/ruby/stdlib/reline/terminfo.rb:40
      <main> at /Users/headius/work/jruby/lib/ruby/stdlib/reline/terminfo.rb:152
     require at org/jruby/RubyKernel.java:1186
     require at /Users/headius/work/jruby/lib/ruby/stdlib/rubygems/core_ext/kernel_require.rb:136
      <main> at /Users/headius/work/jruby/lib/ruby/stdlib/reline.rb:9
     require at org/jruby/RubyKernel.java:1186
     require at /Users/headius/work/jruby/lib/ruby/stdlib/rubygems/core_ext/kernel_require.rb:136
      <main> at -e:1

[headius]

Again, as I said in the original description, this might be a JRuby issue, but I am filing it here for other reasons:

• This is a new native dependency in reline.
• This dependency is not documented or reported during installation of the reline gem.

Perhaps there's a way to make this dependency on libncurses lazy and only load and use it if necessary? It's a separate issue from the inability to load on MacOS (or specifically on my machine), but it is definitely a concern for e.g. minimized Docker containers that may not need libncurses otherwise.

[headius]

JRuby has used "java" for RUBY_PLATFORM for almost twenty years

Another point: RubyGems decided it was better to use RbConfig::CONFIG['arch'] to select an appropriate native extension gem. Why isn't that (or similar) the right way to locate native dependencies in libraries like reline?

[eregon]

What's the path of your libncurses.dylib in Homebrew?
Are you on macos-arm64 or macos-amd64?
If the former, /opt/homebrew/lib should be shown as part of the error message, that's why I thought too old FFI might be the issue.
Maybe some platform detection goes wrong on JRuby? The logic on master is https://github.com/ffi/ffi/blob/c128cede750242fe19945af8bd6c797728489ad5/lib/ffi/dynamic_library.rb#L37-L49

[headius]

ncurses appears to install in /opt/homebrew/Cellar:

$ find /opt/homebrew -name libncurses\*dylib
/opt/homebrew/Cellar/ncurses/6.5/lib/libncurses.6.dylib
/opt/homebrew/Cellar/ncurses/6.5/lib/libncurses++w.6.dylib
/opt/homebrew/Cellar/ncurses/6.5/lib/libncursesw.dylib
/opt/homebrew/Cellar/ncurses/6.5/lib/libncurses.dylib
/opt/homebrew/Cellar/ncurses/6.5/lib/libncursesw.6.dylib
/opt/homebrew/Cellar/ncurses/6.5/lib/libncurses++.dylib
/opt/homebrew/Cellar/ncurses/6.5/lib/libncurses++w.dylib
/opt/homebrew/Cellar/ncurses/6.5/lib/libncurses++.6.dylib

$ brew install ncurses
...
Warning: ncurses 6.5 is already installed and up-to-date.
To reinstall 6.5, run:
  brew reinstall ncurses

I am on macos-aarch64. The SEARCH_PATH does appear to be set up with /opt/homebrew/lib but I do not know why it doesn't show in the error message:

[] json $ jruby -rffi -e 'p FFI::Platform::ARCH'  
"aarch64"
[] json $ jruby -rffi -e 'p FFI::Platform.mac?' 
true
[] json $ jruby -rffi -e 'p FFI::DynamicLibrary::SEARCH_PATH'
["/opt/homebrew/lib", "/opt/local/lib", "/usr/local/lib", "/usr/lib"]

Neither does /opt/local/lib, or /usr/local/lib.

[eregon]

Mmh, so there is no /opt/homebrew/lib/ncurses*.dylib? It seems not with that find output.
Looking at https://github.com/Homebrew/homebrew-core/blob/a320ea608f4eef1318e12c3a5afeaa0a080858f3/Formula/n/ncurses.rb#L22 so it is keg-only and not linked under /opt/homebrew/lib.
The reason is "provided_by_macos", does macOS ship libncurses?

[headius]

There are two issues to address here:

• Finding libncurses dynamic library to be loaded on MacOS with Homebrew. The paths provided by FFI seem insufficient, even if we could confirm they were being honored, since libncurses appears to install in its own Cellar location.
• Depending on ncurses will break other users the same way as it is broken here, even if they do not need the terminfo capabilities it provides. Can those features be optional? (Caveat: I have not looked to see why libncurses is needed).

[headius]

The reason is "provided_by_macos", does macOS ship libncurses?

I have not been able to find it in the usual places, but macOS sprinkles libraries all over the place.

[headius]

macOS sprinkles libraries all over the place

I have been unable to locate libncurses.dylib anywhere in the standard macOS dirs.

This may be related (.tbd as a new dylib extension for "text-based stub libraries"): https://discourse.cmake.org/t/newer-versions-of-macos-require-linking-against-tbd-files-and-not-old-system-paths-to-dylib/6871/2

I do have some libncurses*.tbd files in various places.

[headius]

tbd files for libncurses on my machine. Note that without xcode these would not exist either. I can find no evidence that macOS provides libncurses in any reliable way (and there seems to be some debate about this elsewhere online too):

/Library/Developer/CommandLineTools/SDKs/MacOSX13.3.sdk/usr/lib/libncurses.5.tbd
/Library/Developer/CommandLineTools/SDKs/MacOSX13.3.sdk/usr/lib/libncurses.5.4.tbd
/Library/Developer/CommandLineTools/SDKs/MacOSX13.3.sdk/usr/lib/libncurses.tbd
/Library/Developer/CommandLineTools/SDKs/MacOSX14.4.sdk/usr/lib/libncurses.5.tbd
/Library/Developer/CommandLineTools/SDKs/MacOSX14.4.sdk/usr/lib/libncurses.5.4.tbd
/Library/Developer/CommandLineTools/SDKs/MacOSX14.4.sdk/usr/lib/libncurses.tbd
/System/Volumes/Data/Library/Developer/CommandLineTools/SDKs/MacOSX13.3.sdk/usr/lib/libncurses.5.4.tbd
/System/Volumes/Data/Library/Developer/CommandLineTools/SDKs/MacOSX13.3.sdk/usr/lib/libncurses.tbd

[eregon]

@headius If you look at https://github.com/ruby/reline/blob/master/lib/reline/terminfo.rb libcurses is clearly optional.
You're getting a LoadError above, but it should be rescued by
https://github.com/ruby/fiddle/blob/1f818e46843965fbbb085114f0b875c3acf87489/lib/fiddle/ffi_backend.rb#L478-L479

[eregon]

Ah it's a bug of that code:

@lib = FFI::DynamicLibrary.open(libname, flags) rescue LoadError

is the same as

@lib = begin
  FFI::DynamicLibrary.open(libname, flags)
rescue
  LoadError
end

I'll fix it in ruby/fiddle: ruby/fiddle#156

[tompng]

Thank you for addressing the fiddle issue.

In Reline, libncurses is optional, and in most case that environment variable TERM is set to xterm or other frequently used values, Reline does not need libncurses at all.
So another possible solution for this would be completely drop libncurses dependency. #769

[headius]

@tompng Yes thank you for your response! I can confirm that there's no issue with libncurses anymore for me (on aarch64 macOS) using fiddle 1.1.4.

This issue can be closed. I would love to see #769 happen but for my purposes the fiddle fix is good enough.