Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WIP]: Updates for Pyodide builds after pyodide-build was unvendored #2002

Draft
wants to merge 31 commits into
base: main
Choose a base branch
from

Conversation

agriyakhetarpal
Copy link

@agriyakhetarpal agriyakhetarpal commented Sep 11, 2024

Tip

This PR is currently a work in progress and is not yet ready for review. However, feedback is welcome. The text below is a stub and will be updated with code changes.

This PR updates the Pyodide build procedure (see #1456) that is enabled with CIBW_PLATFORM: "pyodide" (or with the --platform pyodide CLI equivalent) post the changes in pyodide/pyodide#4882, where pyodide/pyodide-build was unvendored from the main Pyodide repository to accommodate faster updates and fixes.

This means that the Pyodide version and pyodide-build are not going to be in sync going forward, and that the Pyodide xbuildenv to install must be inferred by the versions available to install by pyodide-build through a recently added pyodide xbuildenv search command, which prints out this table:

Tap to expand table
Starting new HTTPS connection (1): raw.githubusercontent.com:443
https://raw.githubusercontent.com:443 "GET /pyodide/pyodide/main/pyodide-cross-build-environments.json HTTP/11" 200 917
Version         Python          Emscripten      pyodide-build                   Compatible
----------      ----------      ----------      -------------------------       ----------
0.27.0a2        3.12.1          3.1.58          0.26.0 -                        Yes       
0.26.2          3.12.1          3.1.58          0.26.0 -                        Yes       
0.26.1          3.12.1          3.1.58          0.26.0 -                        Yes       
0.26.0          3.12.1          3.1.58          0.26.0 -                        Yes   

Alternatively, one may use pyodide xbuildenv search --all to return both compatible and non-compatible versions. This would, however, be better received as JSON (please see pyodide/pyodide-build#28).


Additionally, in this PR, I would like to implement support for installing arbitrary Pyodide versions, or, more specifically, arbitrary Pyodide xbuildenv versions – though, only the ones that are supported for a given pyodide-build version. This could be done through an environment variable PYODIDE_XBUILDENV_VERSION (or, PYODIDE_VERSION because it's shorter) and an associated configuration variable in the schema. It would therefore be great to get the above table in machine-readable code to validate it inside cibuildwheel/pyodide.py – for which I opened pyodide/pyodide-build#26 and I'm working on that right now. The rationale behind this is that WebAssembly/Pyodide builds are already experimental anyway, and it would be useful to not tie the available Pyodide version to the cibuildwheel version – this would be helpful for downstream projects (statsmodels/statsmodels#9343, scikit-image/scikit-image#7525, scikit-learn/scikit-learn#29791, and so on) to allow testing against Pyodide alpha releases and/or for the case of reproducibility against Pyodide's older releases.

cc: @hoodmane and @ryanking13 for visibility

@agriyakhetarpal agriyakhetarpal marked this pull request as draft September 11, 2024 13:18
@agriyakhetarpal
Copy link
Author

The Windows test failures are unrelated. I'll try to fix them later in the day, but happy to step back if someone else does it before me, or wishes to.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant