Installing CPAN modules for a custom Perl install with installprefix different from prefix

[vadimkantorov]

I'm building a custom Perl install which includes:

1. a Perl with all modules built statically
2. CPAN-installed modules, including build time dependences for (3) - (having to install using cpan after Perl build because of Perl 5.40.0 hangs forever during building when certain extras packages are specified #22786)
3. Some custom-built modules - which I need to be built statically, so having to extract from tarballs and build them statically manually

After steps (1)-(2)-(3) my custom Perl build should be complete and not modified any further.

Another peculiarity is that I'm building this Perl build in order to make it available at run time in some path like /opt/myperlprefix (but this path is not available at build time), so I thought I'd use -Dprefix=/opt/myperlprefix and -Dinstallprefix set to some path existing at build time.

(1) passes well, but (2) fails badly. The first problem is that (2) tries to use prefix and does not use at all installprefix for module search, so I had to provide PERLLIB and CPATH to make installprefix discoverable. But then it produces NOT OK and nasty warning messages during the modules' build time.

If I don't use a separate installprefix != prefix, all works well.

How could I properly prepare such a perl install? E.g. if I don't use a separate installprefix, how can I redefine INC after the install? (i.e. I'd like that after my build setup and perl install is complete, only /opt/myperlprefix is used for module search path) If it's not possible, how to properly run cpan to make the Perl build fully discoverable without sketchy patches using PERLLIB / CPATH? (and currently cpan produces NOT OK anyway)

Thanks!

log_cpan.txt

My GitHub Actions workflow file, using Ubuntu and Alpine:

name: bug

on: workflow_dispatch
env:
  PERLVER: "5.40.0"
  URLPERL: https://www.cpan.org/src/5.0/perl-5.40.0.tar.gz

jobs:
  bug:
    runs-on: ubuntu-20.04
    container: alpine:3.14
    steps:
      - name: Install Prerequisites
        run:  apk add --update --no-cache libnsl libnsl-dev build-base coreutils gdb cmake git xz curl gperf p7zip zip autoconf automake libtool pkgconfig gnupg libxml2-dev libxslt-dev expat-dev openssl-dev openssl  zlib-static expat-static wget

      - uses: actions/checkout@v4

      - name: Install Perl static
        shell: bash
        run: |
          export PERLINSTALLPREFIX=$RUNNER_WORKSPACE/myperlinstallprefix
          export PERLPREFIX=/opt/myperlprefix
          #export PERLPREFIX=$PERLINSTALLPREFIX # starts with /__w/
          export LD_LIBRARY_PATH=$PERLINSTALLPREFIX/lib:$LD_LIBRARY_PATH

          mkdir perlsourcestatic && curl -L $URLPERL | tar -xzf - --strip-components=1 --directory=perlsourcestatic
          export BUILD_ZLIB=0
          cd perlsourcestatic && sh +x ./Configure -sde -Dman1dir=none -Dman3dir=none -Dprefix="$PERLPREFIX" -Dinstallprefix="$PERLINSTALLPREFIX" -Dusedevel -Uversiononly -Dlibs="-lpthread -ldl -lm -lutil -lc /lib/libz.a" -Dstatic_ext="mro Devel/Peek File/DosGlob File/Glob Sys/Syslog Sys/Hostname PerlIO/via PerlIO/mmap PerlIO/encoding B attributes Unicode/Normalize Unicode/Collate threads threads/shared IPC/SysV re Digest/MD5 Digest/SHA SDBM_File Math/BigInt/FastCalc Data/Dumper I18N/Langinfo Time/HiRes Time/Piece IO Socket Hash/Util/FieldHash Hash/Util Filter/Util/Call POSIX Encode/Unicode Encode Encode/JP Encode/KR Encode/EBCDIC Encode/CN Encode/Symbol Encode/Byte Encode/TW Compress/Raw/Zlib Compress/Raw/Bzip2 MIME/Base64 Cwd Storable List/Util Fcntl Opcode" && cd -
          make -C perlsourcestatic
          make -C perlsourcestatic install

          export PERLLIB=$PERLINSTALLPREFIX/lib/$PERLVER:$PERLINSTALLPREFIX/lib/$PERLVER/x86_64-linux:$PERLINSTALLPREFIX/lib/site_perl/$PERLVER
          export CPATH=$PERLINSTALLPREFIX/lib/$PERLVER/x86_64-linux/CORE
          # added Config::AutoConf to the next line and now it fails with NOT OK
          $PERLINSTALLPREFIX/bin/perl $PERLINSTALLPREFIX/bin/cpan -T Alien::Base::Wrapper Alien::Build Alien::Build::MM Alien::cmake3 Alien::Libxml2 inc::Module::Install Module::Implementation Config::AutoConf

[Grinnz]

See https://github.com/Perl/perl5/blob/blead/INSTALL#changing-the-installation-directory and the following section, in case they are useful

[vadimkantorov]

Yeah, that's where I learned of -Dinstallprefix and -Dprefix differences, but it appears that for any further cpan installs it's not working well. So I'm wondering how to do it properly (so that I could still use cpan and manual module installs, and then switch the install to use only /opt/myperlprefix which I don't have available at build time, also because I don't want to assume sudo rights and also because I'd like that my custom perl binary does not consider any other system-provided perl installs)

I also cannot use -Duserelocatableinc, as at run time I'd have only the /opt/myperlprefix/lib available and the perl binary itself will be run from a different place. This is also the setup for the emscripten Perl I built in #22875

Is INC burned into the perl binary itself? (or e.g. in libperl.a / libperl.so?) Or where is it stored in the Perl install? How can I make sure that the final INC includes only my custom installprefix?

[Leont]

Yes, if files aren't installed to the place where they're expected to be installed, lots of things break. In particular it seems that in this case it can't find the header files. AFAICT it's all working as advertised.

I don't quite understand why you're using installprefix here in the first place. What are you trying to achieve by doing so?

Is INC burned into the perl binary itself?

Yes.

[vadimkantorov]

What are you trying to achieve by doing so?

Because I wanted to use prefix which is not available at build-time (e.g. /opt/myperlprefix). Essentially I have two phases: first build of perl and installation of modules (which can use -Dprefix=$HOME/mycustomperlprefix) and then the second build of perl (which should use @INC based only on /opt/myperlprefix)

Yes.

Is it burned into perlmain.o or into libperl.a? E.g. when I want to build my final perl binary (sometimes using existing perlmain.o, sometimes building a custom obj using https://perldoc.perl.org/perlembed), how can I ensure that @INC only contains /opt/myperlprefix

Basically I'd like to use some accessible prefix during modules install (like /home/myuser/mycustomperprefix), and then at the final binary build time some inaccessible prefix (like /opt/myperlprefix)

[Leont]

Is it burned into perlmain.o or into libperl.a?

It's burned into libperl.

Basically I'd like to use some accessible prefix during modules install (like /home/myuser/mycustomperprefix), and then at the final binary build time some inaccessible prefix (like /opt/myperlprefix)

Why would you want that?

[vadimkantorov]

Why would you want that?

I'm building a custom fully static app which embeds a Perl interpreter (manually packing biber essentially https://github.com/busytex/busybiber/blob/main/.github/workflows/busybiber.yml) and shipping inside the binary all the files via a virtual file system. So I wanted to make sure that this Perl never even tries to use any host Perl install's prefix and instead better errors out if it cannot find the required modules in the virtual file system (as it likely indicates an error on my part).

[vadimkantorov]

I also wonder, is it possible to do proper module installs using cpan (instead of using forever-hanging -Dextras) before calling make install on the Perl's Makefile?

[Leont]

So I wanted to make sure that this Perl never even tries to use any host Perl install's prefix and instead better errors out if it cannot find the required modules in the virtual file system (as it likely indicates an error on my part).

Then just make sure the paths aren't the same? It really sounds like you're making your own life harder than it needs to be.

I also wonder, is it possible to do proper module installs using cpan (instead of using forever-hanging -Dextras) before calling make install on the Perl's Makefile?

Pure perl might actually work, I don't think XS will. I'm not sure it's something we should want to support.

[vadimkantorov]

Then just make sure the paths aren't the same?

This is what I tried to do with my initial attempt to use prefix != installprefix. But this unfortunately breaks further Perl install preparation (i.e. installation of extra modules)...

Pure perl might actually work, I don't think XS will.

Any example of calling cpan before make install? Would the Makefile install anything I put into ext dir?. This might be the solution to my problem, even without XS, as I'm building a fully static build, so I'll need to rebuild the final Perl binary anyway and can provide the XS libs paths myself.

[vadimkantorov]

Which object file in libperl.a contains the value of INC burned in? Maybe I could manually rebuild/replace this object? Thanks!

[mauke]

Probably perl.o. There is code in S_init_perllib that adds entries to @INC.