Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Filter discovered Python executables by source before querying #11143

Merged
merged 3 commits into from
Jan 31, 2025
Merged

Conversation

zanieb
Copy link
Member

@zanieb zanieb commented Jan 31, 2025

Closes #11138

Though I think we could still have a better error message there.

@zanieb zanieb added performance Potential performance improvement error messages Messaging when something goes wrong labels Jan 31, 2025
@zanieb zanieb force-pushed the zb/test-install-inc branch from c9c7b81 to 626248f Compare January 31, 2025 19:49
.chain(from_virtual_environments)
.chain(from_base_conda_environment)
.chain(from_installed)
.filter(move |result| {
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a filter_ok in #11145

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this doing more work than before, since we're returning all the iterators here?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Uhh maybe. I'd have to squint at it a bit.

Generally, it's much less work. This is pre-query (and querying is the slow part) and EnvironmentPreference::ExplicitSystem is the common case so more aggressive filtering here should be better.

It seems possible with EnvironmentPreference::OnlyVirtual we now traverse the PATH directories whereas previously we did not? I'll check on that — because I agree that's not ideal. I'd be sad about doing both this filtering and the previous match statement though, they overlap significantly.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes you're correct, this now scans more in the OnlyVirtual case. However... we only construct that case in uv-dev (and tests)

EnvironmentPreference::OnlyVirtual,

which was surprising to me. We basically always use this constructor

impl EnvironmentPreference {
pub fn from_system_flag(system: bool, mutable: bool) -> Self {
match (system, mutable) {
// When the system flag is provided, ignore virtual environments.
(true, _) => Self::OnlySystem,
// For mutable operations, only allow discovery of the system with explicit selection.
(false, true) => Self::ExplicitSystem,
// For immutable operations, we allow discovery of the system environment
(false, false) => Self::Any,
}
}
}

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The other case is that we'll now also perform the scanning for executables from virtual environments if you provide --system, though we'll toss those out. Might be worth changing still.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

e.g.

main

❯ uv python find -v --system
DEBUG uv 0.5.26+6 (ca5b84027 2025-01-31)
DEBUG Found project root: `/Users/zb/workspace/uv`
DEBUG Project `uv` is marked as unmanaged
DEBUG Reading Python requests from version file at `/Users/zb/workspace/uv/.python-version`
DEBUG Using Python request `3.12` from version file at `.python-version`
DEBUG Searching for Python 3.12 in managed installations or search path
DEBUG Searching for managed installations at `/Users/zb/.local/share/uv/python`
DEBUG Skipping incompatible managed installation `cpython-3.14.0a4-macos-aarch64-none`
DEBUG Skipping incompatible managed installation `cpython-3.10.16-macos-aarch64-none`
DEBUG Skipping incompatible managed installation `cpython-3.10.5-macos-aarch64-none`
DEBUG Found `cpython-3.12.8-macos-aarch64-none` at `/opt/homebrew/bin/python3.12` (search path)
/opt/homebrew/opt/[email protected]/bin/python3.12

vs branch

❯ uv python find -v --system
DEBUG uv 0.5.26+7 (9aefd76b4 2025-01-31)
DEBUG Found project root: `/Users/zb/workspace/uv`
DEBUG Project `uv` is marked as unmanaged
DEBUG Reading Python requests from version file at `/Users/zb/workspace/uv/.python-version`
DEBUG Using Python request `3.12` from version file at `.python-version`
DEBUG Searching for Python 3.12 in managed installations or search path
DEBUG Ignoring Python interpreter at `/Users/zb/workspace/uv/.venv/bin/python3`: system interpreter required
DEBUG Searching for managed installations at `/Users/zb/.local/share/uv/python`
DEBUG Skipping incompatible managed installation `cpython-3.14.0a4-macos-aarch64-none`
DEBUG Skipping incompatible managed installation `cpython-3.10.16-macos-aarch64-none`
DEBUG Skipping incompatible managed installation `cpython-3.10.5-macos-aarch64-none`
DEBUG Found `cpython-3.12.8-macos-aarch64-none` at `/opt/homebrew/bin/python3.12` (search path)
/opt/homebrew/opt/[email protected]/bin/python3.12

Copy link
Member Author

@zanieb zanieb Jan 31, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for flagging 02e72be

Base automatically changed from zb/test-install-inc to main January 31, 2025 20:54
@zanieb zanieb enabled auto-merge (squash) January 31, 2025 21:45
@zanieb zanieb merged commit ba8504f into main Jan 31, 2025
73 checks passed
@zanieb zanieb deleted the zb/filter branch January 31, 2025 21:54
tmeijn pushed a commit to tmeijn/dotfiles that referenced this pull request Feb 4, 2025
This MR contains the following updates:

| Package | Update | Change |
|---|---|---|
| [astral-sh/uv](https://github.com/astral-sh/uv) | patch | `0.5.25` -> `0.5.27` |

MR created with the help of [el-capitano/tools/renovate-bot](https://gitlab.com/el-capitano/tools/renovate-bot).

**Proposed changes to behavior should be submitted there as MRs.**

---

### Release Notes

<details>
<summary>astral-sh/uv (astral-sh/uv)</summary>

### [`v0.5.27`](https://github.com/astral-sh/uv/blob/HEAD/CHANGELOG.md#0527)

[Compare Source](astral-sh/uv@0.5.26...0.5.27)

##### Enhancements

-   Avoid setting permissions during tar extraction ([#&#8203;11191](astral-sh/uv#11191))
-   Remove warnings for missing lower bounds ([#&#8203;11195](astral-sh/uv#11195))
-   Update PubGrub to set-based outdated priority tracking ([#&#8203;11169](astral-sh/uv#11169))
-   Improve error messages for `uv pip install` with `--extra` or `--all-extras` and invalid sources ([#&#8203;11193](astral-sh/uv#11193))
-   Sign Docker images using GitHub attestations ([#&#8203;8685](astral-sh/uv#8685))

##### Preview features

-   Don't expand self-referential extras in the build backend ([#&#8203;11142](astral-sh/uv#11142))

##### Performance

-   Filter discovered Python executables by source before querying ([#&#8203;11143](astral-sh/uv#11143))
-   Optimize exclusion computation for markers ([#&#8203;11158](astral-sh/uv#11158))
-   Use Astral-maintained `tokio-tar` fork ([#&#8203;11174](astral-sh/uv#11174))
-   Remove unneeded `.clone()` ([#&#8203;11127](astral-sh/uv#11127))

##### Bug fixes

-   Fix relative paths in bytecode compilation ([#&#8203;11177](astral-sh/uv#11177))
-   Percent-decode URLs in canonical comparisons ([#&#8203;11088](astral-sh/uv#11088))
-   Respect concurrency limits in parallel index fetch ([#&#8203;11182](astral-sh/uv#11182))
-   Use wire JSON schema for conflict items ([#&#8203;11196](astral-sh/uv#11196))
-   Use explicit `_GLibCVersion` tuple in uv-python crate ([#&#8203;11122](astral-sh/uv#11122))

##### Documentation

-   Add Git SHA locking behavior to docs ([#&#8203;11125](astral-sh/uv#11125))
-   Add best-practice flags to `pip install` example in troubleshooting guide ([#&#8203;11194](astral-sh/uv#11194))
-   Set `VIRTUAL_ENV` in Jupyter kernels ([#&#8203;11155](astral-sh/uv#11155))
-   Add instructions for deactivating an environment ([#&#8203;11200](astral-sh/uv#11200))

### [`v0.5.26`](https://github.com/astral-sh/uv/blob/HEAD/CHANGELOG.md#0526)

[Compare Source](astral-sh/uv@0.5.25...0.5.26)

##### Enhancements

-   Add support for `uvx python` ([#&#8203;11076](astral-sh/uv#11076))
-   Allow `--no-dev --invert` in `uv tree` ([#&#8203;11068](astral-sh/uv#11068))
-   Update `uv python install --reinstall` to reinstall all previous versions ([#&#8203;11072](astral-sh/uv#11072))
-   Consistently write log messages with capitalized first word ([#&#8203;11111](astral-sh/uv#11111))
-   Suggest `--build-backend` when `--backend` is passed to `uv init` ([#&#8203;10958](astral-sh/uv#10958))
-   Improve retry trace message ([#&#8203;11108](astral-sh/uv#11108))

##### Performance

-   Remove unnecessary UTF-8 conversion in hash parsing ([#&#8203;11110](astral-sh/uv#11110))

##### Bug fixes

-   Ignore non-hash fragments in HTML API responses ([#&#8203;11107](astral-sh/uv#11107))
-   Avoid resolving symbolic links when querying Python interpreters ([#&#8203;11083](astral-sh/uv#11083))
-   Avoid sharing state between universal and non-universal resolves ([#&#8203;11051](astral-sh/uv#11051))
-   Error when `--script` is passing a non-PEP 723 script ([#&#8203;11118](astral-sh/uv#11118))
-   Make metadata deserialization failures non-fatal in the cache ([#&#8203;11105](astral-sh/uv#11105))
-   Mark metadata as dynamic when reading from built wheel cache ([#&#8203;11046](astral-sh/uv#11046))
-   Propagate credentials for `<index>/simple` to `<index>/...` endpoints ([#&#8203;11074](astral-sh/uv#11074))
-   Fix conflicting extra bug during `uv sync` ([#&#8203;11075](astral-sh/uv#11075))

##### Documentation

-   Add PyTorch XPU instructions to the PyTorch guide ([#&#8203;11109](astral-sh/uv#11109))
-   Add docs for signal handling ([#&#8203;11041](astral-sh/uv#11041))
-   Explain build frontend vs. build backend ([#&#8203;11094](astral-sh/uv#11094))
-   Fix formatting of `RUST_LOG` documentation ([#&#8203;10053](astral-sh/uv#10053))
-   Fix typo in `--no-deps` description ([#&#8203;11073](astral-sh/uv#11073))
-   Reflow CLI documentation comments ([#&#8203;11040](astral-sh/uv#11040))
-   Shorten "Using existing Python versions" nav item so it fits on one line ([#&#8203;11077](astral-sh/uv#11077))
-   Some minor touch-ups to the Python install guide ([#&#8203;11116](astral-sh/uv#11116))
-   Update Dependabot tracking issue link ([#&#8203;11054](astral-sh/uv#11054))
-   Update documentation for running in a container ([#&#8203;11052](astral-sh/uv#11052))
-   Upgrade PyTorch version in documentation ([#&#8203;11114](astral-sh/uv#11114))
-   Use `sys_platform` in lieu of `platform_system` in PyTorch docs ([#&#8203;11113](astral-sh/uv#11113))
-   Use positive (rather than negative) markers in PyTorch examples ([#&#8203;11112](astral-sh/uv#11112))
-   Fix unnecessary backslashes in brackets ([#&#8203;11059](astral-sh/uv#11059))
-   Suggest setting copy link mode in GitLab integration guide ([#&#8203;11067](astral-sh/uv#11067))

</details>

---

### Configuration

📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).

🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied.

♻ **Rebasing**: Whenever MR becomes conflicted, or you tick the rebase/retry checkbox.

🔕 **Ignore**: Close this MR and you won't be reminded about this update again.

---

 - [ ] <!-- rebase-check -->If you want to rebase/retry this MR, check this box

---

This MR has been generated by [Renovate Bot](https://github.com/renovatebot/renovate).
<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzOS4xNDMuMCIsInVwZGF0ZWRJblZlciI6IjM5LjE1Ni4xIiwidGFyZ2V0QnJhbmNoIjoibWFpbiIsImxhYmVscyI6WyJSZW5vdmF0ZSBCb3QiXX0=-->
zanieb added a commit that referenced this pull request Feb 4, 2025
…nd on the `PATH` (#11218)

Closes #11214

Special-cases the first Python executable we find on the `PATH`,
allowing it to be considered during searches for virtual environments.

For some context, there are two stages to Python interpreter discovery

1. We find possible Python executables in various sources
2. We query the executables to determine canonical metadata about the
interpreter

We can't really be "sure" if an executable is a complaint virtual
environment during (1), we need to query the interpreter first. This
means that if you're only allowed to installed into virtual
environments, we'll query every interpreter on your PATH. This is not
performant, and causes confusion for users. Notably, I recently improved
error messaging when we can't find any valid interpreters, by showing
the error message we encounter while querying an interpreter (if any).
However, this is problematic when there's an error for an interpreter
that is not relevant to your search. In
#11143, I added filtering to avoid
querying additional interpreters, but that regressed some user
experiences where they were relying on us finding implicitly active
virtual environments via the PATH.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
error messages Messaging when something goes wrong performance Potential performance improvement
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants