WIP: Introduce spack-fairsoft extension #311

dennisklein · 2020-09-01T14:19:43Z

An attempt to improve usability:

$ spack fairsoft -h
usage: spack fairsoft [-h] SUBCOMMAND ...

manage FairSoft distros

positional arguments:
  SUBCOMMAND
    avail     show available FairSoft distros
    info      show info on a FairSoft distro
    install   install a FairSoft distro
    list      list installed FairSoft distros
    setup     setup spack for FairSoft
    view      manage symlink spack views on a FairSoft distro

optional arguments:
  -h, --help  show this help message and exit

ideas:

spack fairsoft avail - list available distros

$ spack fairsoft avail
==> 5 distros
    dev.sim
    dev.sim_mt
    dev.sim_mt_headless
    jun19.sim
    jun19.sim_mt

spack fairsoft install <distro> - creates/installs a named environment and tracks the git revision to know when to recreate the environment
spack fairsoft list - basically a filtered spack env list, maybe with an additional column where the view is

$ spack fairsoft list
==> 3 distros
    dev.sim       [121/132 installed packages]  linux-fedora32-skylake  [email protected]
    jun19.sim     [130 installed packages]      linux-fedora32-skylake  [email protected]
    jun19.sim_mt  [127/130 installed packages]  linux-fedora32-skylake  [email protected]

spack fairsoft view [--no-fairroot] [--prefix <prefix>] <distro> - mainly wrapper around spack view which hides details and platform-specific handling from user
spack fairsoft setup [--check-only] - move some of the thisfairsoft.sh --setup business to the python domain

$ spack fairsoft setup
==> Found no new compilers
==> Compilers are defined in the following files:
    /home/dklein/projects/FairSoft2/spack/etc/spack/compilers.yaml
==> Removing all temporary build stages
==> Removing cached information on repositories
==> Report problems at https://github.com/FairRootGroup/FairSoft/issues/new

$ spack fairsoft setup -h
usage: spack fairsoft setup [-h] [--skip-clean] [--skip-compiler] [--skip-git]
                            [--skip-repos] [--config-dir CONFIG_DIR]

setup spack for FairSoft

optional arguments:
  -h, --help            show this help message and exit
  --skip-clean          do not run `spack clean -ms`
  --skip-compiler       do not run `spack compiler find --scope site`
  --skip-git            do not run `git submodule update --init`
  --skip-repos          do not configure repos
  --config-dir CONFIG_DIR
                        symlink provided config dir

Tasks performed:
1. `git submodule update --init`
2. `spack compiler find --scope site`
3. Manage fairsoft site config dir
4. Check/Update repo config
5. `spack clean -ms`

Do not run in parallel to another running spack command!

spack fairsoft info <distro> - print some infos on a distro (we could add a env/<release>/<variant>/info.py for stuff we cannot compute)

$ spack fairsoft info dev.sim
Distro:   dev.sim

Description:
    FairRoot v18.4.0 + full simulation dependencies (single-threaded
    Geant4)

Release: dev

Variant: sim

Packages: TODO

Prerequisites installed: TODO

Installed: TODO

spack fairsoft doctor [--report] - does a number of sanity checks and reports problems, --report option can be used to generate some output to be posted with github bug reports to conveniently describe the user system/setup.
spack fairsoft check_prereqs <distro>

~~@ChristianTackeGSI I would be interested in an early comment if this goes in a direction you would like to see too.~~

Example:

git clone https://github.com/FairRootGroup/FairSoft && cd FairSoft
. ./thisfairsoft.sh --setup
spack fairsoft install jun19.sim
spack fairsoft view jun19.sim /some/dir

ChristianTackeGSI

Yes, I like the idea!
Creating extensions felt way more complex from the docs…

Details maybe need a bit of discussion. Although I don't have any concrete points.

thisfairsoft.sh

extensions/spack-fairsoft/fairsoft/cmd/fairsoft.py

dennisklein · 2020-09-01T16:54:56Z

Details maybe need a bit of discussion. Although I don't have any concrete points.

Ill go ahead and implement a bit more, so we get some more meat to discuss. Thx for the comments so far.

ChristianTackeGSI

Haven't tried it. Looks good!

ChristianTackeGSI · 2020-09-03T15:56:04Z

extensions/spack-fairsoft/fairsoft/__init__.py

+    Return True if an update was performed.
+    """
+    actual = config.get('repos', scope=scope) or []
+    if not actual == expected:


Haven't tried it. But what happens, if user adds some private repos to the site config?

Iff we don't support that, then we should put an error about that

Iff we do, we should probably iteratte over "expected" and check, that it is already in the current config.

Yep, just the first iteration - for now, user can opt-out with --skip-repos.

If we have good ideas, we should definitely make this smarter. What makes this non-trivial is that the config sections do not override repo lists but chain. And order is important here!

ChristianTackeGSI · 2020-09-03T17:00:11Z

One note:
The spack in VAE might be too old for some of this. (Especially, the config add is probably missing). So take a little care when touching thisvae.sh.

Or we "freeze" thisvae.sh now?

dennisklein · 2020-09-03T17:53:54Z

One note:
The spack in VAE might be too old for some of this. (Especially, the config add is probably missing). So take a little care when touching thisvae.sh.

Or we "freeze" thisvae.sh now?

For now I do not plan to change it in this PR. IMHO, we should discuss and come up with a strategy for VAE spack packaging first.

ChristianTackeGSI

Looks good to me.

I haven't tried the setup stuff. But if it works in the CI, then we're already good to go.

We can improve after that.

ChristianTackeGSI · 2020-09-22T11:38:59Z

extensions/spack-fairsoft/fairsoft/__init__.py

+    actual = RepoPath(*(config.get('repos') or []))
+
+    res = False
+    if not actual == expected:


This comparison tooling needs improvements.

Main use case: Panda adding their own repos (say in the user scope).

Agreed. For now there is the opt-out flag --skip-repos until we have something smarter.

extensions/spack-fairsoft/fairsoft/__init__.py

* Add extension skeleton with * first implementations of * spack fairsoft avail * spack fairsoft setup

dennisklein · 2020-11-01T09:09:17Z

Closing, we will re-visit in Q2/21.

dennisklein requested a review from ChristianTackeGSI September 1, 2020 14:19

ChristianTackeGSI reviewed Sep 1, 2020

View reviewed changes

thisfairsoft.sh Outdated Show resolved Hide resolved

extensions/spack-fairsoft/fairsoft/cmd/fairsoft.py Outdated Show resolved Hide resolved

extensions/spack-fairsoft/fairsoft/cmd/fairsoft.py Outdated Show resolved Hide resolved

dennisklein force-pushed the fairsoft_ext branch from 57f7389 to 5cc767a Compare September 1, 2020 16:49

dennisklein force-pushed the fairsoft_ext branch from 5cc767a to 19489d5 Compare September 1, 2020 23:38

dennisklein requested a review from ChristianTackeGSI September 1, 2020 23:51

ChristianTackeGSI approved these changes Sep 3, 2020

View reviewed changes

dennisklein linked an issue Sep 11, 2020 that may be closed by this pull request

Docs section "IV.1. Spack view" #301

Open

dennisklein added this to the sep20 milestone Sep 16, 2020

dennisklein force-pushed the fairsoft_ext branch from 19489d5 to 32ce768 Compare September 21, 2020 07:22

ChristianTackeGSI approved these changes Sep 22, 2020

View reviewed changes

dennisklein force-pushed the fairsoft_ext branch from 32ce768 to 739e658 Compare September 25, 2020 11:08

dennisklein changed the title ~~Introduce spack-fairsoft extension~~ WIP: Introduce spack-fairsoft extension Sep 25, 2020

dennisklein force-pushed the fairsoft_ext branch 2 times, most recently from f454f93 to 0f8b147 Compare September 29, 2020 13:40

dennisklein added 6 commits November 1, 2020 09:05

Introduce spack-fairsoft extension

4082b14

* Add extension skeleton with * first implementations of * spack fairsoft avail * spack fairsoft setup

fairsoft-ext: Implement info subcommand

7c214bc

fairsoft-ext: Implement install subcommand

8895548

fairsoft-ext: Factor out distro env name rule

2f2631a

fairsoft-ext: Implement list subcommand

a1bcc88

fairsoft-ext: Increase log verbosity

8683a24

dennisklein force-pushed the fairsoft_ext branch from 0f8b147 to 8683a24 Compare November 1, 2020 09:07

dennisklein closed this Nov 1, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

WIP: Introduce spack-fairsoft extension #311

WIP: Introduce spack-fairsoft extension #311

dennisklein commented Sep 1, 2020 •

edited

Loading

ChristianTackeGSI left a comment

dennisklein commented Sep 1, 2020

ChristianTackeGSI left a comment

ChristianTackeGSI Sep 3, 2020

dennisklein Sep 3, 2020

ChristianTackeGSI commented Sep 3, 2020

dennisklein commented Sep 3, 2020

ChristianTackeGSI left a comment

ChristianTackeGSI Sep 22, 2020

dennisklein Sep 25, 2020

dennisklein commented Nov 1, 2020

WIP: Introduce spack-fairsoft extension #311

WIP: Introduce spack-fairsoft extension #311

Conversation

dennisklein commented Sep 1, 2020 • edited Loading

ChristianTackeGSI left a comment

Choose a reason for hiding this comment

dennisklein commented Sep 1, 2020

ChristianTackeGSI left a comment

Choose a reason for hiding this comment

ChristianTackeGSI Sep 3, 2020

Choose a reason for hiding this comment

dennisklein Sep 3, 2020

Choose a reason for hiding this comment

ChristianTackeGSI commented Sep 3, 2020

dennisklein commented Sep 3, 2020

ChristianTackeGSI left a comment

Choose a reason for hiding this comment

ChristianTackeGSI Sep 22, 2020

Choose a reason for hiding this comment

dennisklein Sep 25, 2020

Choose a reason for hiding this comment

dennisklein commented Nov 1, 2020

dennisklein commented Sep 1, 2020 •

edited

Loading