Tracking Issue for feature(packed_bundled_libs)

[petrochenkov]

This is a tracking issue for the fix for issue "Linking modifier whole-archive and link time cfg are not supported on bundled static libraries" (#99429).
The feature gate for the issue is #![feature(packed_bundled_libs)].

Currently to combine +bundle native libraries with +whole-archive or cfg a new representation of bundled native libraries in a rlib used, in which the library is packed into the archive as a whole (and wrapped into an object file for technical reasons), and then unpacked when you need to link to it.

Other representations are also possible, e.g. keeping the bundled lib in the old "unpacked" format as a set of object files, and adding some description file telling which object files belong to which library, it would be more complex though.

It may be quite reasonable to support both packed and unpacked representations, both have their uses

• the packed representation minimizes differences with classic non-bundled (-bundle) static libraries
• the unpacked representation is needed when the rlib actually provides missing symbols to the native static lib, not just depends on it, as can be seen from the breakage report in [WIP] Native library linking changes for crater testing #102832

Stabilizing the +bundle,+whole-archive combination doesn't need specifying any of these details, we just need to promise that it somehow works.

About tracking issues

Tracking issues are used to record the overall progress of implementation.
They are also used as hubs connecting to other relevant issues, e.g., bugs or open design questions.
A tracking issue is however not meant for large scale discussion, questions, or bug reports about a feature.
Instead, open a dedicated issue for the specific matter and add the relevant feature gate label.

Steps

• Implement the RFC (cc @rust-lang/XXX -- can anyone write up mentoring
  instructions?)
• Adjust documentation (see instructions on rustc-dev-guide)
• Stabilization PR (see instructions on rustc-dev-guide)

Unresolved Questions

None yet.

Implementation history

• Implemented in Enable new rlib in non stable cases #105601.

[petrochenkov]

Future directions:

• We currently have an unstable option -Zpacked-bundled-libs that switches representation of all bundled static native libs to packed, it should be changed to a native lib modifier +/-packed because it's something that can and should be controllable in a more fine-grained way.
• +packed representation doesn't require the bundled file to be a valid archive, so it's actually a good opportunity to support bundling non-archived object files directly into rlibs, like

  #[link(name = "myobject.o", modifiers="+packed,+verbatim", /* etc */)]

  , we just need to avoid wrapping the packed file into an object file it it's already an object file.

[petrochenkov]

So I think we need to eventually switch everything to using the new (packed) format, e.g. on 2024 edition, because it's a slightly breaking change.

I think the main goal is to fully preserve static library semantic, in that case

• all the modifiers specific to static libraries will work
• crates providing both bundling a library and searching it on the system as alternatives will work identically in both cases
• overrides like -Zlink-native-libraries or -Zlink-directives will work without changes if some bundled library is replaced by a library from the system

For dealing with the breakage in #102832 we need:

• Some kind of attribute (or non-attribute) #[requires_symbol(foo)] that is put on foreign blocks and says that the linked library requires some symbol to be provided by downstream.
• Support for bundling object files into rlibs (basically already exists, needs minor tweaks Tracking Issue for feature(packed_bundled_libs) #108081 (comment)), often we don't need libraries but just need to compile some C files and put the resulting objects into our rlib (example - compiler-builtins).

[Be-ing]

We have an interesting use case for +bundle,+whole-archive in CXX-Qt. We have a crate cxx-qt-build which is used by downstream build.rs scripts. Under the hood, cxx-qt-build generates and compiles C++ code (using the cc crate) that contains static variables that aren't referenced from within main (for several purposes, including Qt's resource system and QML plugin registration), so that code needs to be linked with +whole-archive to be included in the final staticlib/bin. Typically cxx-qt-build is used in a single staticlib crate, which works fine (except on Rust 1.69 due to #110912 which was fixed in 1.70). However, we'd like to be able to support having multiple rlib crates using cxx-qt-build in the Rust dependency graph. A proof-of-concept of such an arragement (KDAB/cxx-qt#598) fails to build with:

error: link modifiers combination `+bundle,+whole-archive` is unstable when generating rlibs

I'm not sure how the discussion above about packed versus unpacked rlibs relates to this. Somehow, we need a way to link a library (or object files) built by the cc crate in build.rs to an rlib crate and have those symbols included in the final downstream staticlib and bin products produced by Cargo.

[petrochenkov]

I'm not sure how the discussion above about packed versus unpacked rlibs relates to this.

"Packed" supports +bundle,+whole-archive, "unpacked" do not.

The discussion above is mainly about enabling "packed" by default for everything.
For the +bundle,+whole-archive and +bundle, cfg(...) cases we mostly can just stabilize it, from what I remember.

[Be-ing]

For the +bundle,+whole-archive and +bundle, cfg(...) cases we mostly can just stabilize it, from what I remember.

Do you mean automatically turning on packed_bundled_libs when +whole-archive and +bundle are used together?

[petrochenkov]

@Be-ing
Yes.
This case is not affected by any backward compatibility issues because it produced errors previously.