Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[bazel,python] update rules_python and vendor in requirements.bzl #26090

Closed
wants to merge 2 commits into from
Closed

[bazel,python] update rules_python and vendor in requirements.bzl #26090

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
7ea4040
[bazel] update rules_python to 0.39.0 to match master
timothytrippel Jan 31, 2025
ea8620c
[bazel,python] vendor in requirements.bzl
timothytrippel Jan 30, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion WORKSPACE
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ load("//third_party/python:deps.bzl", "python_deps")
python_deps()
load("//third_party/python:pip.bzl", "pip_deps")
pip_deps()
load("@ot_python_deps//:requirements.bzl", install_ot_python_deps="install_deps")
load("//third_party/python:requirements.bzl", install_ot_python_deps="install_deps")
install_ot_python_deps()

# Google/Bazel dependencies. This needs to be after Python initialization
Expand Down
19 changes: 16 additions & 3 deletions doc/contributing/sw/adding_python_depedencies.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,8 +1,21 @@
# Adding Python Dependencies to the Project
# Python Dependency Management

To comply with software supply chain security requirements of various project partner organizations, we pin our Python packages to specific versions, and provide hashes for each dependency (including transitive dependencies).
We accomplish this with the help of the `pip-compile` tool, which is part of the `pip-tools` package.

Additionally, to make it easier for downstream Bazel projects to dependend on this Bazel project (`@lowrisc_opentitan`), we vendor in the autogenerated `requirements.bzl` file produced by `rules_python`, see the `rules_python` [documentation](https://rules-python.readthedocs.io/en/0.39.0/pypi-dependencies.html#vendoring-the-requirements-bzl-file) for more details.

# Adding Python Dependencies to the Project
If you need to add another Python package to the project, do so by:
1. adding the package and version number to the `python-requirements.in` file, in the form of `<package>==<version>`, and
1. run the script `util/sh/scripts/gen-python-requirements.sh`, which will auto-generate the updated `python-requirements.txt` file.
1. adding the package and version number to the `python-requirements.in` file, in the form of `<package>==<version>`,
1. running the script `util/sh/scripts/gen-python-requirements.sh`, which will auto-generate the updated `python-requirements.txt` file as shown below, and
1. running the `//third_party/python:vendor_requirements` Bazel target which will update the vendored in `requirements.bzl` file as shown below.

```sh
# Update autogenerated python package management files.
./util/sh/scripts/gen-python-requirements.sh > python-requirements.txt
bazel run //third_party/python:vendor_requirements

# Check that the package management files are up to date.
bazel test //third_party/python/...
```
2 changes: 1 addition & 1 deletion hw/ip/otbn/dv/otbnsim/sim/BUILD
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
# SPDX-License-Identifier: Apache-2.0

load("@rules_python//python:defs.bzl", "py_library")
load("@ot_python_deps//:requirements.bzl", "requirement")
load("//third_party/python:requirements.bzl", "requirement")

package(default_visibility = ["//visibility:public"])

Expand Down
2 changes: 1 addition & 1 deletion hw/ip/otbn/util/BUILD
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
# SPDX-License-Identifier: Apache-2.0

load("@rules_python//python:defs.bzl", "py_binary")
load("@ot_python_deps//:requirements.bzl", "requirement")
load("//third_party/python:requirements.bzl", "requirement")

package(default_visibility = ["//visibility:public"])

Expand Down
2 changes: 1 addition & 1 deletion hw/ip/rom_ctrl/util/BUILD
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
# SPDX-License-Identifier: Apache-2.0

load("@rules_python//python:defs.bzl", "py_binary", "py_library", "py_test")
load("@ot_python_deps//:requirements.bzl", "requirement")
load("//third_party/python:requirements.bzl", "requirement")

package(default_visibility = ["//visibility:public"])

Expand Down
2 changes: 1 addition & 1 deletion rules/scripts/BUILD
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
# SPDX-License-Identifier: Apache-2.0

load("@rules_python//python:defs.bzl", "py_test")
load("@ot_python_deps//:requirements.bzl", "requirement")
load("//third_party/python:requirements.bzl", "requirement")

package(default_visibility = ["//visibility:public"])

Expand Down
2 changes: 1 addition & 1 deletion sw/device/silicon_creator/lib/sigverify/sigverify_tests/BUILD
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
# SPDX-License-Identifier: Apache-2.0

load("//rules:autogen.bzl", "autogen_cryptotest_header")
load("@ot_python_deps//:requirements.bzl", "requirement")
load("//third_party/python:requirements.bzl", "requirement")
load(
"//rules/opentitan:defs.bzl",
"fpga_params",
Expand Down
2 changes: 1 addition & 1 deletion sw/device/silicon_creator/lib/sigverify/sphincsplus/test/BUILD
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ load(
"opentitan_test",
"verilator_params",
)
load("@ot_python_deps//:requirements.bzl", "requirement")
load("//third_party/python:requirements.bzl", "requirement")

package(default_visibility = ["//visibility:public"])

Expand Down
2 changes: 1 addition & 1 deletion sw/device/tests/crypto/BUILD
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ load(
"silicon_params",
"verilator_params",
)
load("@ot_python_deps//:requirements.bzl", "requirement")
load("//third_party/python:requirements.bzl", "requirement")

package(default_visibility = ["//visibility:public"])

Expand Down
2 changes: 1 addition & 1 deletion sw/device/tests/crypto/testvectors/wycheproof/BUILD
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@

package(default_visibility = ["//visibility:public"])

load("@ot_python_deps//:requirements.bzl", "requirement")
load("//third_party/python:requirements.bzl", "requirement")

py_binary(
name = "rsa_3072_verify_parse_testvectors",
Expand Down
2 changes: 1 addition & 1 deletion sw/host/cryptotest/testvectors/parsers/BUILD
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
# SPDX-License-Identifier: Apache-2.0

load("@rules_python//python:defs.bzl", "py_binary", "py_library", "py_test")
load("@ot_python_deps//:requirements.bzl", "requirement")
load("//third_party/python:requirements.bzl", "requirement")

package(default_visibility = ["//visibility:public"])

Expand Down
2 changes: 1 addition & 1 deletion sw/host/provisioning/orchestrator/src/BUILD
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
# Licensed under the Apache License, Version 2.0, see LICENSE for details.
# SPDX-License-Identifier: Apache-2.0

load("@ot_python_deps//:requirements.bzl", "requirement")
load("//third_party/python:requirements.bzl", "requirement")
load("@rules_python//python:defs.bzl", "py_binary", "py_library")

package(default_visibility = ["//visibility:public"])
Expand Down
2 changes: 1 addition & 1 deletion sw/host/provisioning/orchestrator/tests/BUILD
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
# Licensed under the Apache License, Version 2.0, see LICENSE for details.
# SPDX-License-Identifier: Apache-2.0

load("@ot_python_deps//:requirements.bzl", "requirement")
load("//third_party/python:requirements.bzl", "requirement")
load("@rules_python//python:defs.bzl", "py_test")
load(
"//sw/device/silicon_creator/manuf/base:provisioning_inputs.bzl",
Expand Down
58 changes: 58 additions & 0 deletions third_party/python/BUILD
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,66 @@
# Licensed under the Apache License, Version 2.0, see LICENSE for details.
# SPDX-License-Identifier: Apache-2.0

load("@bazel_skylib//rules:build_test.bzl", "build_test")
load("@bazel_skylib//rules:diff_test.bzl", "diff_test")
load("@bazel_skylib//rules:write_file.bzl", "write_file")
load("//third_party/python:requirements.bzl", "all_data_requirements", "all_requirements", "all_whl_requirements")

package(default_visibility = ["//visibility:public"])

exports_files([
"gen_requirements.sh",
])

# The requirements.bzl file is using the hub repo to access packages via the
# `requirement` macro and when the requirements.bzl is vendored, the hub
# repo won't be present. As a result, we have to adjust the label scheme in
# the requirements.bzl to make sure that they continue to work.
genrule(
name = "requirements_bzl",
srcs = ["@ot_python_deps//:requirements.bzl"],
outs = ["requirements.clean.bzl"],
cmd = " | ".join([
"cat $<",
# Substitute the name of the hub to ensure that the dependencies do
# not require the hub repo initialized in the WORKSPACE.
"sed -e 's/ot_python_deps/ot_python_deps_vendored/g'",
# Change the labels from using the hub repo to using the spoke repos
# directly.
"sed -e 's|//\\([^:]*\\):pkg|_\\1//:pkg|g'",
"sed -e 's|//\\([^:]*\\):whl|_\\1//:whl|g'",
"sed -e 's|//\\([^:]*\\):data|_\\1//:data|g'",
# Change the convenience macros to use the same naming.
"sed -e 's|//{}:{}|_{}//:{}|g' >$@",
]),
Comment on lines +24 to +36
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a cool way of building this pipeline while being able to intersperse comments.

FYI, I think you could do this with a single sed invocation if you wanted to:

sed \
    -e  's|...|...|g' \
    -e  's|...|...|g' \
    -e  's|...|...|g' \
    $< >$@

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah good idea. Let me update this in a follow up.

)

write_file(
name = "gen_update",
out = "update.sh",
content = [
"#!/usr/bin/env bash",
"cd $BUILD_WORKSPACE_DIRECTORY",
"cp -fv bazel-bin/third_party/python/requirements.clean.bzl third_party/python/requirements.bzl",
],
)

sh_binary(
name = "vendor_requirements",
srcs = ["update.sh"],
data = [":requirements_bzl"],
)

# Similarly ensures that the requirements.bzl file is updated based on the
# requirements.txt lockfile.
diff_test(
name = "test_vendored",
failure_message = "Please run: bazel run //third_party/python:vendor_requirements",
file1 = "requirements.bzl",
file2 = "requirements_bzl",
)

build_test(
name = "test_requirement_lists",
targets = all_requirements + all_whl_requirements + all_data_requirements,
)
11 changes: 8 additions & 3 deletions third_party/python/repos.bzl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,10 +4,15 @@

load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")

_RULES_PYTHON_VERSION = "0.39.0"

def python_repos():
http_archive(
name = "rules_python",
sha256 = "778aaeab3e6cfd56d681c89f5c10d7ad6bf8d2f1a72de9de55b23081b2d31618",
strip_prefix = "rules_python-0.34.0",
url = "https://github.com/bazelbuild/rules_python/releases/download/0.34.0/rules_python-0.34.0.tar.gz",
sha256 = "62ddebb766b4d6ddf1712f753dac5740bea072646f630eb9982caa09ad8a7687",
strip_prefix = "rules_python-{}".format(_RULES_PYTHON_VERSION),
url = "https://github.com/bazelbuild/rules_python/releases/download/{}/rules_python-{}.tar.gz".format(
_RULES_PYTHON_VERSION,
_RULES_PYTHON_VERSION,
),
)
Loading
Loading