Can't run cargo build ---offline with optional dependency

[maximecb]

Problem

We have a situation where we'd really like for our Rust project to be able to build completely offline without accessing crates.io. We'd also like to be able to run cargo test --offline in our GitHub CI workflows. Unfortunately, this doesn't work. We have one single dependency on the capstone library which is clearly marked as optional, and even with the --offline flag, we get this error:

Run cargo test --offline
error: no matching package named `capstone` found
location searched: registry `crates-io`
required by package `yjit v0.1.0 (/home/runner/work/ruby/ruby/yjit)`
As a reminder, you're using offline mode (--offline) which can sometimes cause surprising resolution failures, if this error is too confusing you may wish to retry without the offline flag.

IMO this is clearly a bug. It violates user intuition as to how the --offline switch should behave. The error message even recommends just... Not using the offline switch because it might be too confusing, which is basically telling the user "maybe just don't use this feature, because it's cryptic, and doesn't actually do what you want".

Well, you're right, it is confusing. The dependency is optional, and we haven't enabled the feature requiring it. Therefore there should be no need to fetch said dependency. It really should be that simple.

Steps

1. Create a Cargo.toml file with an optional dependency which is only used by an optional feature.
2. Then run: cargo build --offline

Linking the Cargo.toml file for our project in case someone can spot something wrong with our setup: https://github.com/Shopify/ruby/blob/70fee18c9947ab50b93ec0018f0d6cf4c6c2772f/yjit/Cargo.toml

Possible Solution(s)

Optional dependencies should not be fetched when features requiring them are not enabled. This should be true in general, but especially when the --offline switch is passed.

Notes

I also tried:

• Removing Cargo.lock (doesn't help)
• cargo test --locked --offline (doesn't help)
• cargo test --frozen --offline (doesn't help)

Version

cargo 1.58.0 (f01b232bc 2022-01-19)
release: 1.58.0
commit-hash: f01b232bc7f4d94f0c4603930a5a96277715eb8c
commit-date: 2022-01-19
host: aarch64-apple-darwin
libgit2: 1.3.0 (sys:0.13.23 vendored)
libcurl: 7.77.0 (sys:0.4.51+curl-7.80.0 system ssl:(SecureTransport) LibreSSL/2.8.3)
os: Mac OS 12.2.0 [64-bit]

[memoryruins]

I think this is still due to lockfile generation that was mentioned after #5655 (comment)

lockfile generation requires the assumption that all possible features are enabled. Generating a lockfile without optional features is covered by #5133

[maximecb]

Can you elaborate on how this should be used?

Just cargo build --minimal-cargo-lock --offline ?

I'll go try this and report back, but either way, the error message should probably be changed, and the behavior of --offline is still not user friendly.

[maximecb]

This argument seems not supported for cargo 1.58.0 ?

> cargo build --minimal-cargo-lock --offline
error: Found argument '--minimal-cargo-lock' which wasn't expected, or isn't valid in this context

It's also not seemingly a supported argument for cargo generate-lockfile:

> cargo generate-lockfile --minimal-cargo-lock
error: Found argument '--minimal-cargo-lock' which wasn't expected, or isn't valid in this context

I did some googling and some browsing through the issue you linked and am still not sure how to use --minimal-cargo-lock.

[memoryruins]

The flags in that issue refer to unstable features which would only be available to a nightly cargo. While avoid-dev-deps has been implemented, I do not see --minimal-cargo-lock (or something like it) listed in the reference. It's possible it hasn't been implemented yet.

[maximecb]

I'm new to the Rust community. What's the best way to proceed here?

Are you affiliated with the Rust project? Do you agree that there is in fact a problem with the way the --offline switch is implemented, in that it doesn't behave as expected and doesn't provide a useful error message?

[memoryruins]

I am a user of cargo trying to connect the dots here as well, rather than a member of the team.

Do you agree that there is in fact a problem with the way the --offline switch is implemented, in that it doesn't behave as expected and doesn't provide a useful error message?

Rather than an issue --offline's implementation, my impression is that it's a limitation during lock file generation. Currently generation assumes all features are enabled and that it's platform independent. Other parts of cargo could benefit with changes to these assumptions, such as cargo vendoring dependencies for a single platform.

I agree expanding on the error message could help; perhaps something like, "use cargo fetch on the package before going offline" and a description of potential reasons for "surprising resolution failures".

What's the best way to proceed here?

When asked in #5133 (comment), more design work and time for the team was needed. It has been a couple years since then so the situation could have changed. Let's wait to hear back from the team.

[ehuss]

Yea, that error message is unfortunately confusing. It sounds like you are trying to run --offline without the index downloaded. That is required even for unused optional dependencies, as cargo needs to run dependency resolution to verify that the Cargo.lock is valid.

The error message definitely could be improved here. If it tries to do resolution without an index in offline mode, then it should say that more clearly. And possibly add a recommendation to run cargo fetch before going offline.

[maximecb]

Thank you for the response @ehuss. Good idea to improve the error message and make it more self-explanatory.

What does running cargo fetch do? Would it modify some files so that we can commit the result of cargo fetch into our git repository?

[ehuss]

cargo fetch is used to download the registry index and any registry dependencies in the Cargo.lock file. The files are cached in $CARGO_HOME. I would not recommend checking that into git. If you want to vendor dependencies (to avoid network access), look at cargo vendor. It is possible to check in the vendor directory if you really want to.

[shaver]

I would not recommend checking that into git.

What are the likely bad outcomes of checking this into git, possibly as last-step procedure when preparing a release? The motivation here is to use Rust in a system that requires the ability to build offline, so if the alternative is to commit some dependency-management artifacts then it might be worth enduring a slightly confusing development-time experience for people who start from a tarball of source.

[ssokolow]

What are the likely bad outcomes of checking this into git, possibly as last-step procedure when preparing a release?

I believe they're talking about ~/.cargo/registry, which is equivalent to /var/lib/apt/lists/ on a Debian-family Linux system.

It's not a part of one's specific project repo, but the global metadata store for things that exist on the package repositories you're configured to depend on.

EDIT: It's also over a gigabyte in size.