[Bazel CI] Error in glob: glob pattern 'lib/clang/*/lib/**/*.dylib' didn't match anything

[sgowroji]

CI: https://buildkite.com/bazel/bazel-at-head-plus-downstream/builds/4381#01943ef5-b088-4c02-a5ba-2aa30cdd04f3

Platform: Ubuntu

Logs:

 ERROR: Traceback (most recent call last):
	File "/var/lib/buildkite-agent/.cache/bazel/_bazel_buildkite-agent/e5b85cc96123ab88632bda12b9f1b123/external/toolchains_llvm++llvm+llvm_toolchain_llvm/BUILD.bazel", line 60, column 16, in <toplevel>
		srcs = glob(
Error in glob: glob pattern 'lib/clang/*/lib/**/*.dylib' didn't match anything, but allow_empty is set to False (the default value of allow_empty can be set with --incompatible_disallow_empty_glob).
(04:14:01) ERROR: /var/lib/buildkite-agent/.cache/bazel/_bazel_buildkite-agent/e5b85cc96123ab88632bda12b9f1b123/external/toolchains_llvm++llvm+llvm_toolchain/BUILD.bazel:114:10: Target '@@toolchains_llvm++llvm+llvm_toolchain_llvm//:clang' contains an error and its package is in error and referenced by '@@toolchains_llvm++llvm+llvm_toolchain//:compiler-components-x86_64-linux'
(04:14:01) ERROR: Analysis of target '//:all' failed; build aborted: Analysis failed

Steps:

git clone https://github.com/bazelbuild/examples && cd ../rust-examples/02-hello-cross
git reset b51e3bdd468ce8c4a516d7dca993909dcc84af32  --hard
export USE_BAZEL_VERSION=de4ba2f81c87eeb9a25b6a9777889785f6e1ed7a
bazel build //:all 

CC Greenteam @Wyverald

[meteorcloudy]

@marvin-hansen Are you able to take a look at the test failures of the rust examples with Bazel@HEAD? See the latest build result from https://buildkite.com/bazel/bazel-at-head-plus-downstream/builds/4431#_

If not, we'll have to skip those tests in bazel downstream pipeline.

[marvin-hansen]

Okay, I take a look over the upcoming weekend.

[marvin-hansen]

@meteorcloudy @sgowroji

Okay, I've filled the first PR that renovates Rust example 1-3. For some totally stupid reasons, I managed to carry over the commit history from 6 months ago to the new branch prior to the merge of the current head. This is totally on me, I get it, but on the bright side, these seem to fix some other examples so please take a look if you are able/willing to review the PR.
If it's okay, just squash it together.

[marvin-hansen]

PR's:

Fix for examples 1 - 3: #546
Fix for examples 4 - 7: #547
Fix for examples 8 - 9: #548

If I read the tealeaves correctly, then CI in the last PR is green with all Rust examples passing
meaning, technically, only the last PR needs to be merged to get this issue closed. Either way, please let me know if there is anything else to do.

[meteorcloudy]

The rust examples are still failing with Bazel@HEAD:
https://buildkite.com/bazel/bazel-at-head-plus-downstream/builds/4465#0194f85a-77e0-43a2-a37e-2fdc80b2af93

Error: Failed to parse release version
Caused by:
    unexpected character 'v' while parsing major version number
bazel run failed with exit code 1

I guess this is because the Bazel binary built from HEAD doesn't have a version number?

[marvin-hansen]

Okay, I debug this with Bazel nightly and try to figure out what's happening. Sorry for this but in the past Bazel 8 was really consistent with rules_rust so apparently some major cheese got moved...

[sgowroji]

Here are the latest CI failures in Bazel Examples

[marvin-hansen]

I see, it seems that the generated vendor script can't digest the missing Bazel version number. Let me ask over at the rules_rust slack and see what can be done.

The good news is, all the remaining examples fail for the exact same reason so once that is patched, downstream CI on HEAD should be green again.

[marvin-hansen]

Okay, there are two offending examples: 7 and 8.

For example 8, I resolved the problem by removing the vendoring that causes the build failure.
Please review PR: #554

Example 7 is really tricky b/c it basically showcases the vendoring that generates the script that
causes the failure on Bazel binary built from HEAD. It is indeed a bug in the script generator, but it is not so clear
if / when that gets resolved so I am still investigating a potential workaround.

[marvin-hansen]

The other thing is, I could not replicate the error on nightly
(9.0.0-pre.20250128) so this clearly can only happen on HEAD.

Would it be sensible to exclude example 7 from downstream b/c it really is a quite a corner case?

[marvin-hansen]

The fix for Rust example 8 was merged so I suggest removing example 7 from downstream CI and then to close this issue.
There is no acknowledgement of the Bazel version issue at rules_rust and therefore no chance that the underlying issue will be fixed anytime soon.