Allow setting variables when using system packages

[TimoWilken]

This would allow us to put setup code (e.g. finding the right python executable, etc) in one place in python.sh.

[ktf]

Is this what we discussed yesterday afternoon?

[TimoWilken]

Yes, it is @ktf!

[ktf]

Could you add a testcase for this, so that e.g. it always pick up a different version than the one specified?

[TimoWilken]

Will do!

[ktf]

I was thinking again about this, and I am back to the idea that having a single prefer_system_check which prints out a string which can be used to retrieve the customisations is better. With the current approach you are deemed to start one docker instance / shell per check, every time we run, which is expensive enough to be annoying / slowing down builds.

Moreover, having a single command will remain compatible with older versions of aliBuild (which will simply ignore the additional environment).

So my suggestion goes back to:

prefer_system: <architecture matching>
prefer_system_check: | something which tests for the presence and optionally prints out "alibuild_system_override: <my override version>"
overrides:
  - name: <my override version>
    version: <my override version>
    env:
         ...
  - name: <another override>

[TimoWilken]

@ktf: I've prototyped this in the latest commit.

[ktf]

@TimoWilken I experimented a bit with this in alisw/alidist#5040, but there is a couple of things which probably we should change:

At the moment we rely on SOMETHING_REVISION to decide wether or not something was built using alibuild, however this results in it being set also for the new system packages. Either we drop such variable, or we must allow overriding the recipe as well, e.g. to create a dummy modulefile which integrates with the system detected version of a package. This might actually be a good idea for the cases were we want to detect a specific version of python not available as default yet still installed. E.g. you have both python3.9 and python3.11 and you want to use the former.

Another thing I do not like is that I end up repeating the customization stanzas:

 "Python36+":
    version: "3.6.0"
  "Python311+":
    version: "3.11.0"

maybe the key could be regexp to match on the alibuild_system_override value and it could be used for substitution:

prefer_system_replacement_specs:
  "Python([0-9]*)[.]([0-9]*)":
    version: "system-$1.$2"
    append_path:
       PATH: `brew --prefix python$1.$2`
    recipe: |
       ...create the modulefile...
prefer_system_check: |
  python3 - <<EOF
  import sys; import sqlite3
  if sys.version_info < (3, 5):
    exit(1)
  print("alibuild_system_replace:{}.{}", *sys.version_info)
  EOF
  python3 -m pip --help > /dev/null && printf '#include "pyconfig.h"' | cc -c $(python3-config --includes) -xc -o /dev/null -;
  if [ $? -ne 0 ]; then 
    cat <<EOF
  Python, the Python development packages, and pip must be installed on your system.
  Usually those packages are called python, python-devel (or python-dev) and python-pip.
  EOF
    exit 1;
  fi

[ktf]

Merging this, since it's long due. Let's wait for the end of the HI run before we tag this, though.