Create PyPI source distribution of meson-based setup #39548

tobiasdiez · 2025-02-19T17:07:57Z

Instead of the old setup-tools based source distribution, the new meson-based one is now uploaded to pypi.

Test run: https://github.com/tobiasdiez/sage/actions/runs/13421204553

New sdist: https://github.com/tobiasdiez/sage/actions/runs/13421204553/artifacts/2618948419

📝 Checklist

The title is concise and informative.
The description explains in detail what this PR is about.
I have linked a relevant issue or discussion.
I have created tests covering the changes.
I have updated the documentation and checked the documentation preview.

⌛ Dependencies

github-actions · 2025-02-19T17:58:02Z

Documentation preview for this PR (built with commit 3b28cce; changes) is ready! 🎉
This preview will update shortly after each push to this PR.

tornaria · 2025-02-19T19:01:46Z

Note:

$ du dist/*.tar.gz
30132	dist/sagemath-10.6b6.tar.gz
19720	dist/sagemath_standard-10.6b6.tar.gz

I created these with

python -m build --sdist --no-isolation -o dist .
python -m build --sdist --no-isolation -o dist pkgs/sagemath-standard

To create the sdist meson forced me to install ppl-devel and fplll-devel which should not be necessary, not even to build afaik (they are used via pplpy and fpylll)

tobiasdiez · 2025-02-19T19:55:00Z

Note:

$ du dist/*.tar.gz
30132	dist/sagemath-10.6b6.tar.gz
19720	dist/sagemath_standard-10.6b6.tar.gz

Mhh, personally I find sagemath more intuitive than sagemath-standard (as there is no sagemath-nonstandard). But I guess this will result in issues as sagemath is already occupied by https://pypi.org/project/sagemath/. Can we maybe take over this "project"?

Edit: I've now changed the name in pyproject.toml to sagemath-standard; we can properly rename to sagemath later.

To create the sdist meson forced me to install ppl-devel and fplll-devel which should not be necessary, not even to build afaik (they are used via pplpy and fpylll)

Thanks, ppl is definitely not needed and fplll is only specified for the compilation of linbox_flint_interface. Is this really not needed, even if linbox is compiled with fplll support?

tobiasdiez · 2025-02-19T20:07:49Z

I think the meson build is ready to be the default in this context.

@tornaria @orlitzky @antonio-rojas @isuruf @saraedum Is this working for you? (feel free to ping more people that might make use of the sdist)

tornaria · 2025-02-19T20:18:15Z

Note:
$ du dist/*.tar.gz
30132	dist/sagemath-10.6b6.tar.gz
19720	dist/sagemath_standard-10.6b6.tar.gz
Mhh, personally I find sagemath more intuitive than sagemath-standard (as there is no sagemath-nonstandard). But I guess this will result in issues as sagemath is already occupied by https://pypi.org/project/sagemath/. Can we maybe take over this "project"?

Regardless of pypi, I think it's a good idea to keep sagemath as the name for the sdist. Moreover, I'd be wary of taking over sagemath-standard until this is very well tested since current packaging workflows will need to be changed.

I think the best might be that we keep publishing the sagemath-standard distribution as is, and also publish the sagemath distribution, for a few releases. This gives distros some time and flexibility to switch to the new distribution.

Edit: I've now changed the name in pyproject.toml to sagemath-standard; we can properly rename to sagemath later.

This may lead to confusion. The two distributions are different enough.

To create the sdist meson forced me to install ppl-devel and fplll-devel which should not be necessary, not even to build afaik (they are used via pplpy and fpylll)

Thanks, ppl is definitely not needed and fplll is only specified for the compilation of linbox_flint_interface. Is this really not needed, even if linbox is compiled with fplll support?

No, it's not necessary afaik.

antonio-rojas · 2025-02-19T20:19:07Z

I don't use the sdist, but essentially the only thing blocking me from switching to meson downstream is the missing cli scripts (#39015)

tornaria · 2025-02-19T20:28:32Z

I think the meson build is ready to be the default in this context.

@tornaria @orlitzky @antonio-rojas @isuruf @saraedum Is this working for you? (feel free to ping more people that might make use of the sdist)

The wheels I that get built from this sdist are missing some stuff... I tried

python -m build -n
python -m build -n -o dist pkgs/sagemath-standard

and here's the result

$ du dist/*.whl
70240	dist/sagemath-10.6b6-cp313-cp313-linux_x86_64.whl
327156	dist/sagemath_standard-10.6b6-cp313-cp313-linux_x86_64.whl

Most of the difference (in size) is because sagemath_standard is compiled in debug mode, but still some files are missing: it seems all the .pyx files are not there, and more important there are no binary scripts in the wheel.

How are you testing this? Ideally, one should be able to use python -m build to create the wheel and then python -m installer to install it from the wheel.

EDIT:

$ unzip -l dist/sagemath_standard-10.6b6-cp313-cp313-linux_x86_64.whl | tail -1
1040855629                     5188 files
$ unzip -l dist/sagemath-10.6b6-cp313-cp313-linux_x86_64.whl | tail -1
233318540                     3982 files

Missing files:

$ unzip -l dist/sagemath_standard-10.6b6-cp313-cp313-linux_x86_64.whl | grep "/cython_debug/" | wc -l
574
$ unzip -l dist/sagemath_standard-10.6b6-cp313-cp313-linux_x86_64.whl | grep "\.pyx$" | wc -l
573
$ unzip -l dist/sagemath_standard-10.6b6-cp313-cp313-linux_x86_64.whl | grep "/scripts/" | wc -l 
30

vs

$ unzip -l dist/sagemath-10.6b6-cp313-cp313-linux_x86_64.whl | grep "/cython_debug/" | wc -l
0
$ unzip -l dist/sagemath-10.6b6-cp313-cp313-linux_x86_64.whl | grep "\.pyx$" | wc -l
9
$ unzip -l dist/sagemath-10.6b6-cp313-cp313-linux_x86_64.whl | grep "/scripts/" | wc -l 
0

saraedum · 2025-02-19T21:32:27Z

cc @AndreasEnge I believe you also experienced these missing files when you were trying to build with the standard machinery for guix first.

tornaria · 2025-02-19T23:40:46Z

Also missing:

sagemath_standard-10.6b6.data/data/share/jupyter/kernels/sagemath/kernel.json
sagemath_standard-10.6b6.data/data/share/jupyter/kernels/sagemath/logo-64x64.png
sagemath_standard-10.6b6.data/data/share/jupyter/kernels/sagemath/logo.svg

The kernel argv is currently set to

["/usr/bin/sage", "--python", "-m", "sage.repl.ipython_kernel", "-f", "{connection_file}"]

and I guess it might change to something like

["python", "-m", "sage.repl.ipython_kernel", "-f", "{connection_file}"]

This is similar to what ipykernel does (indeed they use ["python", "-m", "ipykernel_launcher", "-f", "{connection_file}"])

AndreasEnge · 2025-02-20T08:44:01Z

Building the Guix package with python from a git checkout, I am missing the .pyx files in the resulting installation in lib/python3.10/site-packages/sage/*; I am not sure whether this is what you meant, @saraedum .
The kernel.json and logo files are installed into share/jupyter/kernels/sagemath.

tobiasdiez · 2025-02-21T22:50:51Z

Regardless of pypi, I think it's a good idea to keep sagemath as the name for the sdist. Moreover, I'd be wary of taking over sagemath-standard until this is very well tested since current packaging workflows will need to be changed.

I think the best might be that we keep publishing the sagemath-standard distribution as is, and also publish the sagemath distribution, for a few releases. This gives distros some time and flexibility to switch to the new distribution.

@dimpase @sagemath/core could you please try to get https://pypi.org/project/sagemath/ under the sage umbrella? That would be awesome!

tobiasdiez · 2025-02-21T22:57:35Z

Also missing:

sagemath_standard-10.6b6.data/data/share/jupyter/kernels/sagemath/kernel.json
sagemath_standard-10.6b6.data/data/share/jupyter/kernels/sagemath/logo-64x64.png
sagemath_standard-10.6b6.data/data/share/jupyter/kernels/sagemath/logo.svg

Where/how are these files generated? I cannot find them in the repo.

tobiasdiez · 2025-02-21T22:59:57Z

Most of the difference (in size) is because sagemath_standard is compiled in debug mode, but still some files are missing: it seems all the .pyx files are not there, and more important there are no binary scripts in the wheel.

Why would you expect the pyx files to be in the wheel? They are in the sdist, but not needed after the compilation, right?
Forgot about the scripts, they will be added in #39015.

How are you testing this? Ideally, one should be able to use python -m build to create the wheel and then python -m installer to install it from the wheel.

pip install .

antonio-rojas · 2025-02-21T23:35:40Z

Also missing:

sagemath_standard-10.6b6.data/data/share/jupyter/kernels/sagemath/kernel.json
sagemath_standard-10.6b6.data/data/share/jupyter/kernels/sagemath/logo-64x64.png
sagemath_standard-10.6b6.data/data/share/jupyter/kernels/sagemath/logo.svg

Where/how are these files generated? I cannot find them in the repo.

Here

dimpase · 2025-02-22T02:20:14Z

pypi project sagemath is basically a placeholder, owned by Marc Masdeu, @mmasdeu , a number theorist who contributes to Sage.

Hope @mmasdeu won't mind ?

dimpase · 2025-02-22T02:23:59Z

regarding .pyx files in the wheel, aren't they needed for documentation at sage/ipython prompt? Probably in Jupiter too

mmasdeu · 2025-02-22T07:10:45Z

Hope @mmasdeu won't mind ?

I don't mind at all. In fact, if the Sage community prefers that I eliminate the package I will do it. The ecosystem works much differently nowadays than it used to do...

tobiasdiez · 2025-02-28T15:12:58Z

Also missing:

sagemath_standard-10.6b6.data/data/share/jupyter/kernels/sagemath/kernel.json
sagemath_standard-10.6b6.data/data/share/jupyter/kernels/sagemath/logo-64x64.png
sagemath_standard-10.6b6.data/data/share/jupyter/kernels/sagemath/logo.svg

Where/how are these files generated? I cannot find them in the repo.

Here

Thanks! What is the current established best practice for setting up jupyter kernels? Based on https://ipython.readthedocs.io/en/latest/install/kernel_install.html (and similar guides for other kernels) it looks like something:

$ python -m pip install ipykernel
$ python -m ipykernel install [--user] [--name <machine-readable-name>] [--display-name <"User Friendly Name">]

So first install the package (in our case that would be sagemath) and then manually run a command to install the kernel.
Should we adopt this mechanism or continue to always automatically install the kernel during the package installation?

tobiasdiez · 2025-02-28T15:21:06Z

Hope @mmasdeu won't mind ?

I don't mind at all. In fact, if the Sage community prefers that I eliminate the package I will do it. The ecosystem works much differently nowadays than it used to do...

Awesome! Thanks a lot. Could you please organize with @dimpase the transition of the package to the sagemath org?

I've now renamed the meson-package back to sagemath and restored the old sdists. So now the meson and setuptools-based sdists will both published to pypi, under the names sagemath and sagemath-standard, respectively. This should indeed give downstream maintainers the chance to try out the meson-based setup without forcing them to do so. (Although I would like to retire the setuptools based config sooner than later.)

dimpase · 2025-02-28T16:16:34Z

So first install the package (in our case that would be sagemath) and then manually run a command to install the kernel.
Should we adopt this mechanism or continue to always automatically install the kernel during the package installation?

I would keep it separate - not everyone needs jupyter. However, letting the user figure out what to put into the template is not a good idea.
Printing the concrete command to run would be good.

dimpase · 2025-02-28T16:19:11Z

transition of the package to the sagemath org?

is this merely ownership change? @mmasdeu - can you add me and/or @saraedum as the PyPI sagemath package owner?

tornaria · 2025-02-28T17:16:48Z

Based on https://ipython.readthedocs.io/en/latest/install/kernel_install.html

I think this is needed for special cases (i.e. the kernel is installed
on a different site than jupyter because of virtualenvs)

The "default" and simple way, and what ipykernel itself does, is to
add "data" files. Here's how it works:

$ wget -q https://files.pythonhosted.org/packages/source/i/ipykernel/ipykernel-6.29.5.tar.gz
$ sha256sum ipykernel-6.29.5.tar.gz
f093a22c4a40f8828f8e330a9c297cb93dcab13bd9678ded6de8e5cf81c56215  ipykernel-6.29.5.tar.gz
$ tar xf ipykernel-6.29.5.tar.gz 
$ cd ipykernel-6.29.5/
$ python -m build -nw
* Getting build dependencies for wheel...
* Building wheel...
Successfully built ipykernel-6.29.5-py3-none-any.whl
$ unzip -l dist/ipykernel-6.29.5-py3-none-any.whl | grep -F .data
      193  02-02-2020 00:00   ipykernel-6.29.5.data/data/share/jupyter/kernels/python3/kernel.json
     1084  02-02-2020 00:00   ipykernel-6.29.5.data/data/share/jupyter/kernels/python3/logo-32x32.png
     2180  02-02-2020 00:00   ipykernel-6.29.5.data/data/share/jupyter/kernels/python3/logo-64x64.png
     9605  02-02-2020 00:00   ipykernel-6.29.5.data/data/share/jupyter/kernels/python3/logo-svg.svg
$ python -m installer --destdir DESTDIR dist/ipykernel-6.29.5-py3-none-any.whl
$ find DESTDIR/usr/share/jupyter/
DESTDIR/usr/share/jupyter/
DESTDIR/usr/share/jupyter/kernels
DESTDIR/usr/share/jupyter/kernels/python3
DESTDIR/usr/share/jupyter/kernels/python3/logo-64x64.png
DESTDIR/usr/share/jupyter/kernels/python3/logo-32x32.png
DESTDIR/usr/share/jupyter/kernels/python3/logo-svg.svg
DESTDIR/usr/share/jupyter/kernels/python3/kernel.json
$ cat DESTDIR/usr/share/jupyter/kernels/python3/kernel.json
{
 "argv": [
  "python",
  "-m",
  "ipykernel_launcher",
  "-f",
  "{connection_file}"
 ],
 "display_name": "Python 3 (ipykernel)",
 "language": "python",
 "metadata": {
  "debugger": true
 }
}

The json file could be just a static file, for instance:

{
  "argv": [
    "python",
    "-m",
    "sage.repl.ipython_kernel",
    "-f",
    "{connection_file}"
  ],
  "display_name": "SageMath 10.5",
  "language": "sage"
}

dimpase · 2025-02-28T18:54:50Z

one way or another we should support installing the kernel for use in an existing jupyter installation (including remote ones?), not only in the one we vendor.

ATM such support is pretty messy and on the level "maybe try this, it might work" :-(

tornaria · 2025-02-28T20:47:48Z

one way or another we should support installing the kernel for use in an existing jupyter installation (including remote ones?), not only in the one we vendor.

What I described above is to install the kernel for use in the existing jupyter, and it works fine with sagemath-standard. This is the simplest case: it will install the kernel in a location such that a jupyter installed in the same (virtual or not) environment will find it. This is case applies in particular to (most) distribution packaging, and I think it should also apply to sage-the-distro when sagelib and jupyter are both installed in the same venv.

ATM such support is pretty messy and on the level "maybe try this, it might work" :-(

I don't use it a lot, but IME it just works (this is with both sage and jupyter installed from distro packages into /usr prefix).

dimpase · 2025-03-01T05:27:04Z

Hope @mmasdeu won't mind ?

I don't mind at all. In fact, if the Sage community prefers that I eliminate the package I will do it. The ecosystem works much differently nowadays than it used to do...

@mmasdeu - I got and accepted your invite, but "Manage" button on my Projects PyPI page is greyed out for sagemath. I think you should invite me to be an "Owner".

mmasdeu · 2025-03-02T09:24:30Z

I think I added @saraedum as owner and you as a maintainer. I hope that's okay! Marc

…

On Sat, Mar 1, 2025, 6:27 AM Dima Pasechnik ***@***.***> wrote: Hope @mmasdeu <https://github.com/mmasdeu> won't mind ? I don't mind at all. In fact, if the Sage community prefers that I eliminate the package I will do it. The ecosystem works much differently nowadays than it used to do... @mmasdeu <https://github.com/mmasdeu> - I got and accepted your invite, but "Manage" button on my Projects PyPI page is greyed out for sagemath. I think you should invite me to be an "Owner". — Reply to this email directly, view it on GitHub <#39548 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAMQAIT7J5A2OXMJ3AP7WK32SFAL7AVCNFSM6AAAAABXOW5EROVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMOJRHE3DKMRTGA> . You are receiving this because you were mentioned.Message ID: ***@***.***> [image: dimpase]*dimpase* left a comment (sagemath/sage#39548) <#39548 (comment)> Hope @mmasdeu <https://github.com/mmasdeu> won't mind ? I don't mind at all. In fact, if the Sage community prefers that I eliminate the package I will do it. The ecosystem works much differently nowadays than it used to do... @mmasdeu <https://github.com/mmasdeu> - I got and accepted your invite, but "Manage" button on my Projects PyPI page is greyed out for sagemath. I think you should invite me to be an "Owner". — Reply to this email directly, view it on GitHub <#39548 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAMQAIT7J5A2OXMJ3AP7WK32SFAL7AVCNFSM6AAAAABXOW5EROVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMOJRHE3DKMRTGA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

tobiasdiez added 2 commits February 19, 2025 18:06

Migrate PyPI source distribution to use meson

ae6c3e5

Install build package

598b198

tobiasdiez added 2 commits February 19, 2025 20:17

Install python-build instead of build

26fa65b

Run in correct shell

ea06823

Rename project from "sagemath" to "sagemath-standard"

c3a98c0

tobiasdiez marked this pull request as ready for review February 19, 2025 20:07

github-actions bot added the s: needs review label Feb 19, 2025

tobiasdiez mentioned this pull request Feb 21, 2025

Meson: Remove unused build dependencies ppl and fplll #39562

Open

5 tasks

tobiasdiez and others added 3 commits February 28, 2025 16:17

Restore sdist creation of sagemath-standard

104d1ae

Rename meson back to sagemath

5c459f6

Merge branch 'develop' into meson-dist

3b28cce

tobiasdiez changed the title ~~Migrate PyPI source distribution to use meson~~ Create PyPI source distribution of meson-based setup Feb 28, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Create PyPI source distribution of meson-based setup #39548

Create PyPI source distribution of meson-based setup #39548

tobiasdiez commented Feb 19, 2025 •

edited

Loading

github-actions bot commented Feb 19, 2025 •

edited

Loading

tornaria commented Feb 19, 2025

tobiasdiez commented Feb 19, 2025 •

edited

Loading

tobiasdiez commented Feb 19, 2025

tornaria commented Feb 19, 2025

antonio-rojas commented Feb 19, 2025

tornaria commented Feb 19, 2025 •

edited

Loading

saraedum commented Feb 19, 2025

tornaria commented Feb 19, 2025

AndreasEnge commented Feb 20, 2025

tobiasdiez commented Feb 21, 2025

tobiasdiez commented Feb 21, 2025

tobiasdiez commented Feb 21, 2025

antonio-rojas commented Feb 21, 2025

dimpase commented Feb 22, 2025

dimpase commented Feb 22, 2025

mmasdeu commented Feb 22, 2025

tobiasdiez commented Feb 28, 2025

tobiasdiez commented Feb 28, 2025

dimpase commented Feb 28, 2025

dimpase commented Feb 28, 2025

tornaria commented Feb 28, 2025

dimpase commented Feb 28, 2025 •

edited

Loading

tornaria commented Feb 28, 2025

dimpase commented Mar 1, 2025

mmasdeu commented Mar 2, 2025 via email

Create PyPI source distribution of meson-based setup #39548

Are you sure you want to change the base?

Create PyPI source distribution of meson-based setup #39548

Conversation

tobiasdiez commented Feb 19, 2025 • edited Loading

📝 Checklist

⌛ Dependencies

github-actions bot commented Feb 19, 2025 • edited Loading

tornaria commented Feb 19, 2025

tobiasdiez commented Feb 19, 2025 • edited Loading

tobiasdiez commented Feb 19, 2025

tornaria commented Feb 19, 2025

antonio-rojas commented Feb 19, 2025

tornaria commented Feb 19, 2025 • edited Loading

saraedum commented Feb 19, 2025

tornaria commented Feb 19, 2025

AndreasEnge commented Feb 20, 2025

tobiasdiez commented Feb 21, 2025

tobiasdiez commented Feb 21, 2025

tobiasdiez commented Feb 21, 2025

antonio-rojas commented Feb 21, 2025

dimpase commented Feb 22, 2025

dimpase commented Feb 22, 2025

mmasdeu commented Feb 22, 2025

tobiasdiez commented Feb 28, 2025

tobiasdiez commented Feb 28, 2025

dimpase commented Feb 28, 2025

dimpase commented Feb 28, 2025

tornaria commented Feb 28, 2025

dimpase commented Feb 28, 2025 • edited Loading

tornaria commented Feb 28, 2025

dimpase commented Mar 1, 2025

mmasdeu commented Mar 2, 2025 via email

tobiasdiez commented Feb 19, 2025 •

edited

Loading

github-actions bot commented Feb 19, 2025 •

edited

Loading

tobiasdiez commented Feb 19, 2025 •

edited

Loading

tornaria commented Feb 19, 2025 •

edited

Loading

dimpase commented Feb 28, 2025 •

edited

Loading