Skip to content

Making changes to the build environment

Jump to bottom
Joshua Nelson edited this page Dec 21, 2019 · 8 revisions

This is not intended for anything other than testing changes to rustops/crates-build-env. In almost every other scenario, you should use docker-compose or https://github.com/rust-lang/docs.rs/wiki/Developing-without-docker-compose instead.

Rustwide internally uses rustops/crates-build-env as the build environment for the crate. If you want to add a system package for crates to link to, this is place you're looking for.

NOTE: these instructions currently only work on Linux, see https://github.com/rust-lang/docs.rs/issues/522 for a possible workaround on other Unix platforms.

Getting started

First, clone the crates-build-env repo:

$ git clone https://github.com/rust-lang/crates-build-env && cd crates-build-env

Next, add the package to packages.txt. This should be the name of a package in the Ubuntu 18.04 Repositories. See the package home page for a full list/search bar, or use apt search locally.

Building the image

Now build the image. This will take a very long time, probably 10-20 minutes.

$ docker build --tag build-env .

Testing the image

Use the image to build your crate. This command assumes you are in the directory of your crates' source; if not, cd there now. If you have never built the crate before, run cargo check now so that cargo will download your dependencies and generate Cargo.lock.

The following command replicates the environment used on docs.rs precisely. If it builds here, it should build on docs.rs, barring of course any changes you've made to build-env.

$ cd /path/to/your/crate
$ CONTAINER="$("docker" "create" \
    "-v" "$(realpath ./target):/opt/rustwide/target:rw,Z" \
    "-v" "$(realpath .):/opt/rustwide/workdir:ro,Z" \
    "-v" ~/.cargo:/opt/rustwide/cargo-home:ro,Z \
    "-v" ~/.rustup:/opt/rustwide/rustup-home:ro,Z \
    "-e" "SOURCE_DIR=/opt/rustwide/workdir" "-e" "MAP_USER_ID=$(id -u)" \
    "-e" "CARGO_TARGET_DIR=/opt/rustwide/target" "-e" "RUSTFLAGS=" \
    "-e" 'RUSTDOCFLAGS=-Z unstable-options 
        --resource-suffix -20191116-1.41.0-nightly-5c5b8afd8 
        --static-root-path / --disable-per-crate-search' \
    "-e" "CARGO_HOME=/opt/rustwide/cargo-home" \
    "-e" "RUSTUP_HOME=/opt/rustwide/rustup-home" \
    "-w" "/opt/rustwide/workdir" "-m" "3221225472" "--network" "none" \
    "build-env" "/opt/rustwide/cargo-home/bin/cargo" \
        "+nightly" "doc" --offline --locked "--lib" "--no-deps" "--target" "x86_64-unknown-linux-gnu")"
$ docker start -ai "$CONTAINER"

Making multiple changes

If your build fails even after your changes, it will be annoying to rebuild the image from scratch just to add a single package. Instead, you can make changes directly to the Dockerfile so that the existing packages are cached. Be sure to move these new packages from the Dockerfile to packages.txt once you are sure they work.

On line 7 of the Dockerfile, add this line: RUN apt-get install -y your_second_package. Rerun the build and start the container; it should take much less time now:

$ docker build --tag build-env .
$ cd /path/to/your/crate
$ CONTAINER="$("docker" "create" \
    "-v" "$(realpath ./target):/opt/rustwide/target:rw,Z" \
    "-v" "$(realpath .):/opt/rustwide/workdir:ro,Z" \
    "-v" ~/.cargo:/opt/rustwide/cargo-home:ro,Z \
    "-v" ~/.rustup:/opt/rustwide/rustup-home:ro,Z \
    "-e" "SOURCE_DIR=/opt/rustwide/workdir" "-e" "MAP_USER_ID=$(id -u)" \
    "-e" "CARGO_TARGET_DIR=/opt/rustwide/target" "-e" "RUSTFLAGS=" \
    "-e" 'RUSTDOCFLAGS=-Z unstable-options 
        --resource-suffix -20191116-1.41.0-nightly-5c5b8afd8 
        --static-root-path / --disable-per-crate-search' \
    "-e" "CARGO_HOME=/opt/rustwide/cargo-home" \
    "-e" "RUSTUP_HOME=/opt/rustwide/rustup-home" \
    "-w" "/opt/rustwide/workdir" "-m" "3221225472" "--network" "none" \
    "build-env" "/opt/rustwide/cargo-home/bin/cargo" \
        "+nightly" "doc" --offline --locked "--lib" "--no-deps" "--target" "x86_64-unknown-linux-gnu")"
$ docker start -ai "$CONTAINER"

Run the lint script

Before you make a PR, run the shell script ci/lint.sh and make sure it passes. It ensures packages.txt is in order and will tell you exactly what changes you need to make if not.

Make a pull request

Once you are sure your package builds, you can make a pull request to get it adopted upstream for docs.rs and crater. Go to https://github.com/rust-lang/crates-build-env and click 'Fork' in the top right. Locally, add your fork as a remote in git and push your changes:

$ git remote add personal https://github.com/<your_username_here>/crates-build-env
$ git add -u
$ git commit -m 'add packages necessary for <your_package_here> to compile'
$ git push personal

Back on github, make a pull request:

  1. Go to https://github.com/rust-lang/crates-build-env/compare
  2. Click 'compare across forks'
  3. Click 'head repository' -> <your_username>/crates-build-env
  4. Click 'Create pull request'
  5. Add a description of what packages you added and what crate they fixed
  6. Click 'Create pull request' again in the bottom right.

Hopefully your changes will be merged quickly! After that you can either publish a point release (rebuilds your docs immediately) or request for a member of the docs.rs team to schedule a new build (may take a while depending on their schedules).

Clone this wiki locally