-
Notifications
You must be signed in to change notification settings - Fork 1.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Filter discovered Python executables by source before querying #11143
Conversation
c9c7b81
to
626248f
Compare
crates/uv-python/src/discovery.rs
Outdated
.chain(from_virtual_environments) | ||
.chain(from_base_conda_environment) | ||
.chain(from_installed) | ||
.filter(move |result| { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is a filter_ok
in #11145
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is this doing more work than before, since we're returning all the iterators here?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Uhh maybe. I'd have to squint at it a bit.
Generally, it's much less work. This is pre-query (and querying is the slow part) and EnvironmentPreference::ExplicitSystem
is the common case so more aggressive filtering here should be better.
It seems possible with EnvironmentPreference::OnlyVirtual
we now traverse the PATH
directories whereas previously we did not? I'll check on that — because I agree that's not ideal. I'd be sad about doing both this filtering and the previous match
statement though, they overlap significantly.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes you're correct, this now scans more in the OnlyVirtual
case. However... we only construct that case in uv-dev
(and tests)
uv/crates/uv-dev/src/compile.rs
Line 26 in 41cd4be
EnvironmentPreference::OnlyVirtual, |
which was surprising to me. We basically always use this constructor
uv/crates/uv-python/src/discovery.rs
Lines 1616 to 1627 in e26affd
impl EnvironmentPreference { | |
pub fn from_system_flag(system: bool, mutable: bool) -> Self { | |
match (system, mutable) { | |
// When the system flag is provided, ignore virtual environments. | |
(true, _) => Self::OnlySystem, | |
// For mutable operations, only allow discovery of the system with explicit selection. | |
(false, true) => Self::ExplicitSystem, | |
// For immutable operations, we allow discovery of the system environment | |
(false, false) => Self::Any, | |
} | |
} | |
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The other case is that we'll now also perform the scanning for executables from virtual environments if you provide --system
, though we'll toss those out. Might be worth changing still.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
e.g.
main
❯ uv python find -v --system
DEBUG uv 0.5.26+6 (ca5b84027 2025-01-31)
DEBUG Found project root: `/Users/zb/workspace/uv`
DEBUG Project `uv` is marked as unmanaged
DEBUG Reading Python requests from version file at `/Users/zb/workspace/uv/.python-version`
DEBUG Using Python request `3.12` from version file at `.python-version`
DEBUG Searching for Python 3.12 in managed installations or search path
DEBUG Searching for managed installations at `/Users/zb/.local/share/uv/python`
DEBUG Skipping incompatible managed installation `cpython-3.14.0a4-macos-aarch64-none`
DEBUG Skipping incompatible managed installation `cpython-3.10.16-macos-aarch64-none`
DEBUG Skipping incompatible managed installation `cpython-3.10.5-macos-aarch64-none`
DEBUG Found `cpython-3.12.8-macos-aarch64-none` at `/opt/homebrew/bin/python3.12` (search path)
/opt/homebrew/opt/[email protected]/bin/python3.12
vs branch
❯ uv python find -v --system
DEBUG uv 0.5.26+7 (9aefd76b4 2025-01-31)
DEBUG Found project root: `/Users/zb/workspace/uv`
DEBUG Project `uv` is marked as unmanaged
DEBUG Reading Python requests from version file at `/Users/zb/workspace/uv/.python-version`
DEBUG Using Python request `3.12` from version file at `.python-version`
DEBUG Searching for Python 3.12 in managed installations or search path
DEBUG Ignoring Python interpreter at `/Users/zb/workspace/uv/.venv/bin/python3`: system interpreter required
DEBUG Searching for managed installations at `/Users/zb/.local/share/uv/python`
DEBUG Skipping incompatible managed installation `cpython-3.14.0a4-macos-aarch64-none`
DEBUG Skipping incompatible managed installation `cpython-3.10.16-macos-aarch64-none`
DEBUG Skipping incompatible managed installation `cpython-3.10.5-macos-aarch64-none`
DEBUG Found `cpython-3.12.8-macos-aarch64-none` at `/opt/homebrew/bin/python3.12` (search path)
/opt/homebrew/opt/[email protected]/bin/python3.12
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for flagging 02e72be
This MR contains the following updates: | Package | Update | Change | |---|---|---| | [astral-sh/uv](https://github.com/astral-sh/uv) | patch | `0.5.25` -> `0.5.27` | MR created with the help of [el-capitano/tools/renovate-bot](https://gitlab.com/el-capitano/tools/renovate-bot). **Proposed changes to behavior should be submitted there as MRs.** --- ### Release Notes <details> <summary>astral-sh/uv (astral-sh/uv)</summary> ### [`v0.5.27`](https://github.com/astral-sh/uv/blob/HEAD/CHANGELOG.md#0527) [Compare Source](astral-sh/uv@0.5.26...0.5.27) ##### Enhancements - Avoid setting permissions during tar extraction ([#​11191](astral-sh/uv#11191)) - Remove warnings for missing lower bounds ([#​11195](astral-sh/uv#11195)) - Update PubGrub to set-based outdated priority tracking ([#​11169](astral-sh/uv#11169)) - Improve error messages for `uv pip install` with `--extra` or `--all-extras` and invalid sources ([#​11193](astral-sh/uv#11193)) - Sign Docker images using GitHub attestations ([#​8685](astral-sh/uv#8685)) ##### Preview features - Don't expand self-referential extras in the build backend ([#​11142](astral-sh/uv#11142)) ##### Performance - Filter discovered Python executables by source before querying ([#​11143](astral-sh/uv#11143)) - Optimize exclusion computation for markers ([#​11158](astral-sh/uv#11158)) - Use Astral-maintained `tokio-tar` fork ([#​11174](astral-sh/uv#11174)) - Remove unneeded `.clone()` ([#​11127](astral-sh/uv#11127)) ##### Bug fixes - Fix relative paths in bytecode compilation ([#​11177](astral-sh/uv#11177)) - Percent-decode URLs in canonical comparisons ([#​11088](astral-sh/uv#11088)) - Respect concurrency limits in parallel index fetch ([#​11182](astral-sh/uv#11182)) - Use wire JSON schema for conflict items ([#​11196](astral-sh/uv#11196)) - Use explicit `_GLibCVersion` tuple in uv-python crate ([#​11122](astral-sh/uv#11122)) ##### Documentation - Add Git SHA locking behavior to docs ([#​11125](astral-sh/uv#11125)) - Add best-practice flags to `pip install` example in troubleshooting guide ([#​11194](astral-sh/uv#11194)) - Set `VIRTUAL_ENV` in Jupyter kernels ([#​11155](astral-sh/uv#11155)) - Add instructions for deactivating an environment ([#​11200](astral-sh/uv#11200)) ### [`v0.5.26`](https://github.com/astral-sh/uv/blob/HEAD/CHANGELOG.md#0526) [Compare Source](astral-sh/uv@0.5.25...0.5.26) ##### Enhancements - Add support for `uvx python` ([#​11076](astral-sh/uv#11076)) - Allow `--no-dev --invert` in `uv tree` ([#​11068](astral-sh/uv#11068)) - Update `uv python install --reinstall` to reinstall all previous versions ([#​11072](astral-sh/uv#11072)) - Consistently write log messages with capitalized first word ([#​11111](astral-sh/uv#11111)) - Suggest `--build-backend` when `--backend` is passed to `uv init` ([#​10958](astral-sh/uv#10958)) - Improve retry trace message ([#​11108](astral-sh/uv#11108)) ##### Performance - Remove unnecessary UTF-8 conversion in hash parsing ([#​11110](astral-sh/uv#11110)) ##### Bug fixes - Ignore non-hash fragments in HTML API responses ([#​11107](astral-sh/uv#11107)) - Avoid resolving symbolic links when querying Python interpreters ([#​11083](astral-sh/uv#11083)) - Avoid sharing state between universal and non-universal resolves ([#​11051](astral-sh/uv#11051)) - Error when `--script` is passing a non-PEP 723 script ([#​11118](astral-sh/uv#11118)) - Make metadata deserialization failures non-fatal in the cache ([#​11105](astral-sh/uv#11105)) - Mark metadata as dynamic when reading from built wheel cache ([#​11046](astral-sh/uv#11046)) - Propagate credentials for `<index>/simple` to `<index>/...` endpoints ([#​11074](astral-sh/uv#11074)) - Fix conflicting extra bug during `uv sync` ([#​11075](astral-sh/uv#11075)) ##### Documentation - Add PyTorch XPU instructions to the PyTorch guide ([#​11109](astral-sh/uv#11109)) - Add docs for signal handling ([#​11041](astral-sh/uv#11041)) - Explain build frontend vs. build backend ([#​11094](astral-sh/uv#11094)) - Fix formatting of `RUST_LOG` documentation ([#​10053](astral-sh/uv#10053)) - Fix typo in `--no-deps` description ([#​11073](astral-sh/uv#11073)) - Reflow CLI documentation comments ([#​11040](astral-sh/uv#11040)) - Shorten "Using existing Python versions" nav item so it fits on one line ([#​11077](astral-sh/uv#11077)) - Some minor touch-ups to the Python install guide ([#​11116](astral-sh/uv#11116)) - Update Dependabot tracking issue link ([#​11054](astral-sh/uv#11054)) - Update documentation for running in a container ([#​11052](astral-sh/uv#11052)) - Upgrade PyTorch version in documentation ([#​11114](astral-sh/uv#11114)) - Use `sys_platform` in lieu of `platform_system` in PyTorch docs ([#​11113](astral-sh/uv#11113)) - Use positive (rather than negative) markers in PyTorch examples ([#​11112](astral-sh/uv#11112)) - Fix unnecessary backslashes in brackets ([#​11059](astral-sh/uv#11059)) - Suggest setting copy link mode in GitLab integration guide ([#​11067](astral-sh/uv#11067)) </details> --- ### Configuration 📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever MR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this MR and you won't be reminded about this update again. --- - [ ] <!-- rebase-check -->If you want to rebase/retry this MR, check this box --- This MR has been generated by [Renovate Bot](https://github.com/renovatebot/renovate). <!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzOS4xNDMuMCIsInVwZGF0ZWRJblZlciI6IjM5LjE1Ni4xIiwidGFyZ2V0QnJhbmNoIjoibWFpbiIsImxhYmVscyI6WyJSZW5vdmF0ZSBCb3QiXX0=-->
…nd on the `PATH` (#11218) Closes #11214 Special-cases the first Python executable we find on the `PATH`, allowing it to be considered during searches for virtual environments. For some context, there are two stages to Python interpreter discovery 1. We find possible Python executables in various sources 2. We query the executables to determine canonical metadata about the interpreter We can't really be "sure" if an executable is a complaint virtual environment during (1), we need to query the interpreter first. This means that if you're only allowed to installed into virtual environments, we'll query every interpreter on your PATH. This is not performant, and causes confusion for users. Notably, I recently improved error messaging when we can't find any valid interpreters, by showing the error message we encounter while querying an interpreter (if any). However, this is problematic when there's an error for an interpreter that is not relevant to your search. In #11143, I added filtering to avoid querying additional interpreters, but that regressed some user experiences where they were relying on us finding implicitly active virtual environments via the PATH.
Closes #11138
Though I think we could still have a better error message there.