Improve how we manage which PDM to use

[mwoehlke-kitware]

Move our PDM version pin to a separate file. This both makes it easier to manage updates to the same and also allows the pinned version to be usable outside of the venv_sync script. Teach venv_upgrade to use PDM itself to generate this file, which accomplishes the outstanding goal of being able to pin both PDM and its dependencies. Also, modify venv_sync to tell PDM to not complain about newer versions.

Toward #22040.

---

This change is 

[mwoehlke-kitware]

+@BetsyMcPhail for feature review, please.

[BetsyMcPhail]

On our local mac, I ran venv_upgrade with the changes from this PR. It successfully updated both pdm.lock as well as requirements.txt.

Reviewed 5 of 5 files at r1, all commit messages.
Reviewable status: 1 unresolved discussion, needs platform reviewer assigned, needs at least two assigned reviewers, missing label for release notes (waiting on @mwoehlke-kitware)

---

tools/workspace/python/venv_sync line 82 at r1 (raw file):

# Aside from that, we want to use the pinned version, so being informed about
# newer versions is just noise.
export PDM_CHECK_UPDATE=0

BTW The venv_update script still complained about an out-of-date PDM. e.g.

L ./setup/python/pdm.lock
Changes are written to pdm.lock.
0:00:11 🔒 Lock successful.
INFO: PDM 2.22.0 is installed, while 2.22.1 is available.
Please run pdm self update to upgrade.
Run pdm config check_update false to disable the check.

I'm not sure if this is expected with this change? In any case, it did complete the update, so it was just noise.

[mwoehlke-kitware]

Reviewable status: 1 unresolved discussion, needs platform reviewer assigned, needs at least two assigned reviewers, missing label for release notes

---

tools/workspace/python/venv_sync line 82 at r1 (raw file):

Previously, BetsyMcPhail (Betsy McPhail) wrote…

BTW The venv_update script still complained about an out-of-date PDM. e.g.

L ./setup/python/pdm.lock
Changes are written to pdm.lock.
0:00:11 🔒 Lock successful.
INFO: PDM 2.22.0 is installed, while 2.22.1 is available.
Please run pdm self update to upgrade.
Run pdm config check_update false to disable the check.

I'm not sure if this is expected with this change? In any case, it did complete the update, so it was just noise.

Ah. It's "expected" in that only venv_sync has the "please be quiet" logic. However, it's probably sensible if PDM is out of date to update it first. I made a small change to accomplish that (and also noticed a minor error in the process, namely, the repo wasn't monitoring the new requirements.txt for changes); PTAL.

[BetsyMcPhail]

Reviewed 2 of 2 files at r2, all commit messages.
Reviewable status: 1 unresolved discussion, needs platform reviewer assigned, needs at least two assigned reviewers, missing label for release notes (waiting on @mwoehlke-kitware)

---

tools/workspace/python/venv_sync line 82 at r1 (raw file):

Previously, mwoehlke-kitware (Matthew Woehlke) wrote…

Ah. It's "expected" in that only venv_sync has the "please be quiet" logic. However, it's probably sensible if PDM is out of date to update it first. I made a small change to accomplish that (and also noticed a minor error in the process, namely, the repo wasn't monitoring the new requirements.txt for changes); PTAL.

I re-ran venv_update with the latest changes and it completed with no warnings.

[mwoehlke-kitware]

+@xuchenhan-tri for platform review, please.

Reviewable status: LGTM missing from assignee xuchenhan-tri(platform), missing label for release notes (waiting on @mwoehlke-kitware)

[xuchenhan-tri]

@BetsyMcPhail your LGTM stamp is missing.

Platform +(release notes: none)

Reviewed 4 of 5 files at r1, 2 of 2 files at r2, all commit messages.
Reviewable status: 1 unresolved discussion

---

tools/workspace/python/update_requirements.py line 37 at r2 (raw file):

def main(args):

nit, this file needs some brief documentation on what this does.

[mwoehlke-kitware]

On additional poking, there's a slightly better way to do this. Stay tuned. (@BetsyMcPhail, will need a re-review...)

[mwoehlke-kitware]

Done. Note that I am deliberately not updating the requirements.txt in this commit as it is somewhat awkward to do so without also having extraneous version bumps in pdm.lock. Also, this way it is obvious that venv_upgrade actually does something.

[mwoehlke-kitware]

tools/workspace/python/update_requirements.py line 37 at r2 (raw file):

Previously, xuchenhan-tri wrote…

nit, this file needs some brief documentation on what this does.

No longer applicable.

[BetsyMcPhail]

I confirmed venv_upgrade still upgrades with latest changes.

Reviewed 3 of 3 files at r3, all commit messages.
Reviewable status: 1 unresolved discussion

---

tools/workspace/python/venv_upgrade line 59 at r3 (raw file):

readonly requirements="./setup/python/requirements.txt"
cat > "${requirements}" <<EOF
# This file describes does NOT describe the requirements for Drake itself, but

BTW Typo in this sentence

[mwoehlke-kitware]

Reviewable status: complete! all discussions resolved, LGTM from assignees xuchenhan-tri(platform),BetsyMcPhail

---

tools/workspace/python/venv_upgrade line 59 at r3 (raw file):

Previously, BetsyMcPhail (Betsy McPhail) wrote…

BTW Typo in this sentence

Fixed. (Also in setup/python/requirements.txt.)

[mwoehlke-kitware]

@BetsyMcPhail, can you re-check v6 please?

• requirements.txt also gets regenerated after a PDM update.
• The header for the same moved to a .in file instead of being inline in the script.
• There is now a --reuse option to apply pyproject.toml changes while minimizing changes to version pins.

[mwoehlke-kitware]

@drake-jenkins-bot linux-jammy-gcc-bazel-experimental-debug please