From 62ea6335c33e4c9b818cf6d37bdb22ae7c327340 Mon Sep 17 00:00:00 2001 From: alex Date: Mon, 17 Jul 2023 20:11:15 -0500 Subject: [PATCH 01/74] add draft of packaging existing software tutorial --- source/tutorials/index.md | 1 + .../tutorials/packaging-existing-software.md | 870 ++++++++++++++++++ 2 files changed, 871 insertions(+) create mode 100644 source/tutorials/packaging-existing-software.md diff --git a/source/tutorials/index.md b/source/tutorials/index.md index a5f93a1bc..41fe9d838 100644 --- a/source/tutorials/index.md +++ b/source/tutorials/index.md @@ -12,4 +12,5 @@ first-steps/index.md learning-journey/index.md nixos/index.md cross-compilation.md +packaging-existing-software.md ``` diff --git a/source/tutorials/packaging-existing-software.md b/source/tutorials/packaging-existing-software.md new file mode 100644 index 000000000..0f68d1dc7 --- /dev/null +++ b/source/tutorials/packaging-existing-software.md @@ -0,0 +1,870 @@ +--- +myst: + html_meta: + "description lang=en": "Packaging Existing Software With Nix" + "keywords": "Nix, packaging" +--- + + +(packaging-existing-software)= + + +# Packaging Existing Software With Nix + +One of Nix's primary use-cases is in addressing common difficulties encountered while packaging software, like *managing dependencies*. + +In the long term, Nix helps tremendously in alleviating that stress, but when *first* (re)packaging existing software with Nix, it's common to encounter missing dependencies preventing builds from succeeding. + +If you haven't already read the tutorial on making a derivation, please go do so before reading this! + +In this tutorial, we'll see how to create Nix derivations to package C/C++ software. + +We'll begin by considering `hello`, a feature-complete implementation of the famous "hello world", which requires no external dependencies. + +Then we'll move to progressively more complex packages with their own separate dependencies, leading us to use additional derivation features. + +Along the way, we'll encounter and address Nix error messages, build failures, and a host of other issues, developing our iterative debugging techniques as we go. + +Finally, we'll wrap up by preparing these projects for contribution to `nixpkgs`. + +## A Simple Project +To start, we'll write a skeleton derivation, updating this as we go: + +```nix +{ pkgs, stdenv }: +stdenv.mkDerivation { }; +``` + +### Hello, World! +Since GNU Hello is a popular (in a certain sense) package from the GNU Project, we can easily access its source code [from GNU's FTP](https://ftp.gnu.org/gnu/hello/). + +In this case, we'll download the [latest version](https://ftp.gnu.org/gnu/hello/hello-2.12.1.tar.gz) of `hello`, currently `2.12.1`. + +Downloading that into our build context is a good first step; we can do this in several ways, but it's best to use one of the nixpkgs builtin fetcher functions, in this case `fetchTarball` from nixpkgs. + +`fetchTarball` takes as argument an attribute set containing the URI path to the download and a SHA256 hash of the file. + +We won't know what the hash really is until we download it, but fortunately Nix will complain at us if we're wrong, so we can fake it with a bunch of zeroes (53 to be exact): + +```nix +# hello.nix +{ pkgs, stdenv }: +stdenv.mkDerivation { + src = builtins.fetchTarball { + url = "https://ftp.gnu.org/gnu/hello/hello-2.12.1.tar.gz"; + sha256 = "0000000000000000000000000000000000000000000000000000"; + }; +} +``` + +:::{note} +Instead of typing 53 zeroes, you can use `lib.fakeSha256`. We'll do this in later examples. +::: + +Let's save this file to `hello.nix` and try to build it. To do so, we'll use `nix-build`... + +```console +$ nix-build hello.nix +error: cannot evaluate a function that has an argument without a value ('pkgs') + Nix attempted to evaluate a function as a top level expression; in + this case it must have its arguments supplied either by default + values, or passed explicitly with '--arg' or '--argstr'. See + https://nixos.org/manual/nix/stable/language/constructs.html#functions. + + at /home/nix-user/hello.nix:2:3: + + 1| # hello.nix + 2| { pkgs, stdenv }: + | ^ + 3| stdenv.mkDerivation { +``` + +... and immediately run into a problem: every derivation is a *function*, and functions need *arguments*! + +### Your New Favorite Command +In order to pass the `pkgs` argument to our derivation, we'll need to import `nixpkgs` in another Nix expression. The `nix-build` command lets us pass whole expressions as an argument following the `-E/--expr` flag. + +We'll use the following expression: + +```console +with import {}; callPackage ./hello.nix {} +^ ^ ^ ^ ^ ^ ^ +1 2 3 4 5 6 7 +``` + +This expression imports (2) the system `nixpkgs` (3) into the local scope using the `with` statement (1), applying it (remember, like any other derivation in Nix, `nixpkgs` is actually a function!) to an empty attribute set (4). + +From `nixpkgs`, we use the `callPackage` function (5) to import our `hello.nix` (6). Since the `hello.nix` derivation [is a function](https://nixos.org/manual/nix/stable/language/derivations.html) which takes two arguments [it will already be passed](), we also apply it to an empty attribute set (7). Although `hello.nix` takes a `pkgs` argument, we don't need to apply the function to `nixpkgs`, because the `with` statement has already included the `nixpkgs` we just imported into the local context. + +Let's run this now: + +```console +$ nix-build -E 'with import {}; callPackage ./hello.nix {}' +error: derivation name missing +``` +Progress! The new failure occurs with the *derivation*, further down in the file than the initial error on line 2 about the `pkgs` argument not having a value; we successfully resolved the previous error by importing `nixpkgs` in the expression we passed to `nix-build`. + +### Naming a Derivation +Every derivation needs a `name` attribute, which must either be set directly or constructed by `mkDerivation` from `pname` and `version` attributes, if they exist. + +Let's update the file again to add a `name`: + +```nix +{ pkgs, stdenv }: +stdenv.mkDerivation { + name = "hello"; + src = builtins.fetchTarball { + url = "https://ftp.gnu.org/gnu/hello/hello-2.12.1.tar.gz"; + sha256 = "0000000000000000000000000000000000000000000000000000"; + }; +} +``` +and then re-run the command: + +```console +$ nix-build -E 'with import {}; callPackage ./hello.nix {}' +error: + … while calling the 'derivationStrict' builtin + + at /builtin/derivation.nix:9:12: (source not available) + + … while evaluating derivation 'hello' + whose name attribute is located at /nix/store/i6w7hmdjp1jg71g7xbjgz5rn96q443c6-nixos-23.05.1471.b72aa95f7f0/nixos/pkgs/stdenv/generic/make-derivation.nix:303:7 + + … while evaluating attribute 'src' of derivation 'hello' + + at /home/nix-user/hello.nix:5:3: + + 4| name = "hello"; + 5| src = builtins.fetchTarball { + | ^ + 6| url = "https://ftp.gnu.org/gnu/hello/hello-2.12.1.tar.gz"; + + error: hash mismatch in file downloaded from 'https://ftp.gnu.org/gnu/hello/hello-2.12.1.tar.gz': + specified: sha256:0000000000000000000000000000000000000000000000000000 + got: sha256:0xw6cr5jgi1ir13q6apvrivwmmpr5j8vbymp0x6ll0kcv6366hnn +``` + +### Finding The File Hash +As expected, Nix complained at us for lying about the file hash, and helpfully provided the correct one. We can substitute this into our `hello.nix` file, replacing the string of zeroes: + +```nix +# hello.nix +{ pkgs, stdenv }: +stdenv.mkDerivation { + name = "hello"; + src = builtins.fetchTarball { + url = "https://ftp.gnu.org/gnu/hello/hello-2.12.1.tar.gz"; + sha256 = "0xw6cr5jgi1ir13q6apvrivwmmpr5j8vbymp0x6ll0kcv6366hnn"; + }; +} +``` + +Now let's run that command again: + +```console +$ nix-build -E 'with import {}; callPackage ./hello.nix {}' +this derivation will be built: + /nix/store/rbq37s3r76rr77c7d8x8px7z04kw2mk7-hello.drv +building '/nix/store/rbq37s3r76rr77c7d8x8px7z04kw2mk7-hello.drv'... +unpacking sources +unpacking source archive /nix/store/xdbysilxxgbs55rrdxniglqg9m1v61h4-source +source root is source +patching sources +configuring +configure flags: --disable-dependency-tracking --prefix=/nix/store/y55w1djfnxkl2jk9w0liancp83zqb7ki-hello +... +configure: creating ./config.status +config.status: creating Makefile +... +building +build flags: SHELL=/nix/store/7q1b1bsmxi91zci6g8714rcljl620y7f-bash-5.2-p15/bin/bash +... +``` +Great news: our derivation built successfully! + +:::{important} +But how? This looks magical! +::: + +We can see from the console output that `configure` was called, which produced a `Makefile` that was then used to build the project; we didn't actually write any build instructions, so we can surmise that Nix automatically detected the structure of the project directory. Indeed, the build system in Nix is based on `autoconf`. + +### Build Result +We can check our working directory for the result: + +```console +$ ls +hello.nix result +``` + +This result is a symbolic link to a Nix store location containing the built binary; we can call `./result/bin/hello` to execute this program: + +```console +$ ./result/bin/hello +Hello, world! +``` + +We've successfully packaged our first program with Nix! The experience was a little bit *too* magical though, so up next we'll package another piece of software which has external dependencies and a different means of building, which will require us to lean more on `mkDerivation`. + +## Something Bigger +The `hello` program is a simple and common place to start packaging, but it's not very useful or interesting, so we can't stop there. + +Now, we'll look at packaging a somewhat more complicated program, `icat`, which allows us to render images in our terminal. + +Though there are at least two alternative similar tools already in `nixpkgs`, at the time of writing, this particular tool hasn't been packaged, and it is used in {ref}`another tutorial `.) *fixme: complete and merge https://github.com/NixOS/nix.dev/pull/645, then link it here*, so this is a good opportunity to do something both informative and useful. + +We'll start by copying the `hello.nix` from the previous section to a new file, `icat.nix`: + +```nix +# icat.nix +{ pkgs, stdenv }: +stdenv.mkDerivation { + name = "hello"; + src = builtins.fetchTarball { + url = "https://ftp.gnu.org/gnu/hello/hello-2.12.1.tar.gz"; + sha256 = "0xw6cr5jgi1ir13q6apvrivwmmpr5j8vbymp0x6ll0kcv6366hnn"; + }; +} +``` + +While Nix can sometimes feel magic, it's not *actually* magic, so unfortunately this won't magically produce `icat` for us, and we'll need to make several changes. + +To start, we'll need to change the `name` attribute: + +```nix +# icat.nix +{ pkgs, stdenv }: +stdenv.mkDerivation { + name = "icat"; + src = builtins.fetchTarball { + url = "https://ftp.gnu.org/gnu/hello/hello-2.12.1.tar.gz"; + sha256 = "0xw6cr5jgi1ir13q6apvrivwmmpr5j8vbymp0x6ll0kcv6366hnn"; + }; +} +``` + +Now we'll download the source code. `icat`'s upstream repository is hosted on [GitHub](https://github.com/atextor/icat), so we should slightly modify our previous [source fetcher](https://nixos.org/manual/nixpkgs/stable/#chap-pkgs-fetchers): instead of `fetchTarball`, we'll use [`fetchFromGitHub`](https://nixos.org/manual/nixpkgs/stable/#fetchfromgithub): + +```nix +# icat.nix +{ pkgs, stdenv }: +stdenv.mkDerivation { + name = "icat"; + src = builtins.fetchFromGitHub { + url = "https://ftp.gnu.org/gnu/hello/hello-2.12.1.tar.gz"; + sha256 = "0xw6cr5jgi1ir13q6apvrivwmmpr5j8vbymp0x6ll0kcv6366hnn"; + }; +} +``` + +### Fetching Source from GitHub +Rather than `url` and `sha256`, we must now supply the following arguments: +- `owner`: a string representing the user or organization account which owns the repository. The `nixpkgs` repository lives at `https://github.com/NixOS/nixpkgs`, so if we were fetching the `nixpkgs` source, for example, and in this case the source is hosted at `https://github.com/atextor/icat`, so we want `owner = "atextor"`. +- `repo`: another string corresponding to the name of the repository. Here we'll use `repo = "icat"`. +- `rev`: the *revision*, such as the git commit hash or tag (e.g. `v1.0`), to download from GitHub. On GitHub, you can find these on the [Releases page](https://github.com/atextor/icat/releases), but for now we're just going to use `master`. +- `hash`: similar to the earlier `hello` example, this corresponds to the hash of the extracted directory after it is downloaded. Several hash algorithms are available, but we used SHA256 before, so we'll do it again here and use `sha256` in place of the `hash` input. We could fake the hash with 53 zeroes (or any string of characters that doesn't match the *actual* hash) as we did before, but rather than type that out, we'll just use `lib.fakeSha256`. + +Updating our file accordingly: + +```nix +# icat.nix +{ pkgs, stdenv }: +stdenv.mkDerivation { + name = "icat"; + src = builtins.fetchFromGitHub { + owner = "atextor"; + repo = "icat"; + rev = "master"; + sha256 = lib.fakeSha256; + }; +} +``` + +Running our previous `nix-build` invocation: + +```console +$ nix-build -E 'with import {}; callPackage ./icat.nix {}' +error: + … while evaluating a branch condition + + at /nix/store/i6w7hmdjp1jg71g7xbjgz5rn96q443c6-nixos-23.05.1471.b72aa95f7f0/nixos/lib/customisation.nix:179:8: + + 178| + 179| in if missingArgs == [] then makeOverridable f allArgs else abort error; + | ^ + 180| + + … while calling the 'attrNames' builtin + + at /nix/store/i6w7hmdjp1jg71g7xbjgz5rn96q443c6-nixos-23.05.1471.b72aa95f7f0/nixos/lib/customisation.nix:139:21: + + 138| # wouldn't be passed to it + 139| missingArgs = lib.attrNames + | ^ + 140| # Filter out arguments that have a default value + + (stack trace truncated; use '--show-trace' to show the full trace) + + error: undefined variable 'lib' + + at /home/nix-user/icat.nix:9:12: + + 8| rev = "master"; + 9| hash = lib.fakeSha256; + | ^ + 10| }; +``` + +### Namespacing +This one is easy: `lib` lives in the `pkgs` namespace, so we can either fix this by invoking `pkgs.lib.fakeSha256` instead, or by taking `lib` as an argument to the whole expression. The latter option is more common, so we'll do that. This is also a good time to rearrange our set of arguments to better conform to the [Nixpkgs syntactic conventions](https://nixos.org/manual/nixpkgs/stable/#chap-conventions): + +```nix +# icat.nix +{ pkgs +, lib +, stdenv +}: + +stdenv.mkDerivation { + name = "icat"; + src = builtins.fetchFromGitHub { + owner = "atextor"; + repo = "icat"; + rev = "master"; + sha256 = lib.fakeSha256; + }; +} +``` + +```console +$ nix-build -E 'with import {}; callPackage ./icat.nix {}' +error: + … while calling the 'derivationStrict' builtin + + at /builtin/derivation.nix:9:12: (source not available) + + … while evaluating derivation 'icat' + whose name attribute is located at /nix/store/i6w7hmdjp1jg71g7xbjgz5rn96q443c6-nixos-23.05.1471.b72aa95f7f0/nixos/pkgs/stdenv/generic/make-derivation.nix:303:7 + + … while evaluating attribute 'src' of derivation 'icat' + + at /home/nix-user/icat.nix:9:3: + + 8| name = "icat"; + 9| src = builtins.fetchFromGitHub { + | ^ + 10| owner = "atextor"; + + error: attribute 'fetchFromGitHub' missing + + at /home/nix-user/icat.nix:9:9: + + 8| name = "icat"; + 9| src = builtins.fetchFromGitHub { + | ^ + 10| owner = "atextor"; +``` + +Another issue, and the converse of the previous one: `fetchFromGitHub` doesn't live in `builtins`, it lives in `pkgs`: + +```nix +# icat.nix +{ pkgs +, lib +, stdenv +}: + +stdenv.mkDerivation { + name = "icat"; + src = pkgs.fetchFromGitHub { + owner = "atextor"; + repo = "icat"; + rev = "master"; + sha256 = lib.fakeSha256; + }; +} +``` + +We used `lib.fakeSha256`, which does essentially the same thing as our earlier 53 zeroes trick: + +```console +$ nix-build -E 'with import {}; callPackage ./icat.nix {}' +... +unpacking source archive /build/master.tar.gz +error: hash mismatch in fixed-output derivation '/nix/store/lgjf8cq63ahqnd3b117g1q58g4nkprmj-source.drv': + specified: sha256-AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA= + got: sha256-b/2mRzCTyGkz2I1U+leUhspvW77VcHN7Awp+BVdVNRM= +error: 1 dependencies of derivation '/nix/store/afiw4a1l04pi82k6w630d42iflgfxbl6-icat.drv' failed to build +``` + +:::{note} +We've been faking the hash and letting `nix-build` report the correct one in an error, but we could also fetch the correct hash in the first place with one of the `nix-prefetch` commands, or by downloading the tarball and passing it to the appropriate `nix-hash` invocation. We'll use the `nix-prefetch` trick in the next section. +::: + +Now that we have the correct hash, we'll replace `lib.fakeSha256` in the file and re-run the command: + +```nix +# icat.nix +{ pkgs +, lib +, stdenv +}: + +stdenv.mkDerivation { + name = "icat"; + src = pkgs.fetchFromGitHub { + owner = "atextor"; + repo = "icat"; + rev = "master"; + sha256 = "sha256-b/2mRzCTyGkz2I1U+leUhspvW77VcHN7Awp+BVdVNRM="; + }; +} +``` + +### Missing Dependencies +Now we run into an entirely new issue: + +```console +$ nix-build -E 'with import {}; callPackage ./icat.nix {}' +this derivation will be built: + /nix/store/dvqbkap31salw9mbr2xhxnnnb0089x3v-icat.drv +building '/nix/store/dvqbkap31salw9mbr2xhxnnnb0089x3v-icat.drv'... +unpacking sources +unpacking source archive /nix/store/y4750c9xljqy21b62a03z5xqvl3sd92q-source +source root is source +patching sources +configuring +no configure script, doing nothing +building +build flags: SHELL=/nix/store/7q1b1bsmxi91zci6g8714rcljl620y7f-bash-5.2-p15/bin/bash +gcc -c -Wall -pedantic -std=c99 -D_DEFAULT_SOURCE `pkg-config --cflags imlib2` -o icat.o icat.c +/nix/store/7q1b1bsmxi91zci6g8714rcljl620y7f-bash-5.2-p15/bin/bash: line 1: pkg-config: command not found +icat.c:39:10: fatal error: Imlib2.h: No such file or directory + 39 | #include + | ^~~~~~~~~~ +compilation terminated. +make: *** [Makefile:20: icat.o] Error 1 +error: builder for '/nix/store/dvqbkap31salw9mbr2xhxnnnb0089x3v-icat.drv' failed with exit code 2; + last 10 log lines: + > no configure script, doing nothing + > building + > build flags: SHELL=/nix/store/7q1b1bsmxi91zci6g8714rcljl620y7f-bash-5.2-p15/bin/bash + > gcc -c -Wall -pedantic -std=c99 -D_DEFAULT_SOURCE `pkg-config --cflags imlib2` -o icat.o icat.c + > /nix/store/7q1b1bsmxi91zci6g8714rcljl620y7f-bash-5.2-p15/bin/bash: line 1: pkg-config: command not found + > icat.c:39:10: fatal error: Imlib2.h: No such file or directory + > 39 | #include + > | ^~~~~~~~~~ + > compilation terminated. + > make: *** [Makefile:20: icat.o] Error 1 + For full logs, run 'nix log /nix/store/dvqbkap31salw9mbr2xhxnnnb0089x3v-icat.drv'. +``` + +Finally, a compiler error! We've successfully pulled the `icat` source from GitHub, and Nix tried to build what it found, but is missing a dependency: the `imlib2` header. If we [search for `imlib2` on search.nixos.org](https://search.nixos.org/packages?channel=23.05&from=0&size=50&sort=relevance&type=packages&query=imlib2), we'll find that `imlib2` is already in `nixpkgs`. + +We can add this package to our build environment by either +- adding `imlib2` to the set of inputs to the expression in `icat.nix`, and then adding `imlib2` to the list of `buildInputs` in `stdenv.mkDerivation`, or +- adding `pkgs.imlib2` to the `buildInputs` directly, since `pkgs` is already in-scope. + +We'll do the latter of these here: + +```nix +# icat.nix +{ pkgs +, lib +, stdenv +}: + +stdenv.mkDerivation { + name = "icat"; + src = pkgs.fetchFromGitHub { + owner = "atextor"; + repo = "icat"; + rev = "master"; + sha256 = "sha256-b/2mRzCTyGkz2I1U+leUhspvW77VcHN7Awp+BVdVNRM="; + }; + + buildInputs = [ pkgs.imlib2 ]; +} + +``` + +Another error, but we get further this time: + +```console +$ nix-build -E 'with import {}; callPackage ./icat.nix {}' +this derivation will be built: + /nix/store/0csqp747mfw0v9n103abxgx611s6dkxm-icat.drv +building '/nix/store/0csqp747mfw0v9n103abxgx611s6dkxm-icat.drv'... +unpacking sources +unpacking source archive /nix/store/y4750c9xljqy21b62a03z5xqvl3sd92q-source +source root is source +patching sources +configuring +no configure script, doing nothing +building +build flags: SHELL=/nix/store/7q1b1bsmxi91zci6g8714rcljl620y7f-bash-5.2-p15/bin/bash +gcc -c -Wall -pedantic -std=c99 -D_DEFAULT_SOURCE `pkg-config --cflags imlib2` -o icat.o icat.c +/nix/store/7q1b1bsmxi91zci6g8714rcljl620y7f-bash-5.2-p15/bin/bash: line 1: pkg-config: command not found +In file included from icat.c:39: +/nix/store/hkgbjcr182m3q9xs0j1qmp3dh08mbg31-imlib2-1.11.1-dev/include/Imlib2.h:45:10: fatal error: X11/Xlib.h: No such file or directory + 45 | #include + | ^~~~~~~~~~~~ +compilation terminated. +make: *** [Makefile:20: icat.o] Error 1 +error: builder for '/nix/store/0csqp747mfw0v9n103abxgx611s6dkxm-icat.drv' failed with exit code 2; + last 10 log lines: + > building + > build flags: SHELL=/nix/store/7q1b1bsmxi91zci6g8714rcljl620y7f-bash-5.2-p15/bin/bash + > gcc -c -Wall -pedantic -std=c99 -D_DEFAULT_SOURCE `pkg-config --cflags imlib2` -o icat.o icat.c + > /nix/store/7q1b1bsmxi91zci6g8714rcljl620y7f-bash-5.2-p15/bin/bash: line 1: pkg-config: command not found + > In file included from icat.c:39: + > /nix/store/hkgbjcr182m3q9xs0j1qmp3dh08mbg31-imlib2-1.11.1-dev/include/Imlib2.h:45:10: fatal error: X11/Xlib.h: No such file or directory + > 45 | #include + > | ^~~~~~~~~~~~ + > compilation terminated. + > make: *** [Makefile:20: icat.o] Error 1 + For full logs, run 'nix log /nix/store/0csqp747mfw0v9n103abxgx611s6dkxm-icat.drv'. +``` + +In Nixpkgs, `Xlib` lives in the `dev` output of `xorg.libX11`, which we can add to `buildInputs` again with `pkgs.xorg.libX11.dev`. To avoid repeating ourselves, we can add `pkgs` to the local scope in `buildInputs` by using the [`with` statement](https://nixos.org/guides/nix-pills/basics-of-language.html#idm140737320521984): + +```nix +{ pkgs +, lib +, stdenv +}: + +stdenv.mkDerivation { + name = "icat"; + src = pkgs.fetchFromGitHub { + owner = "atextor"; + repo = "icat"; + rev = "master"; + sha256 = "sha256-b/2mRzCTyGkz2I1U+leUhspvW77VcHN7Awp+BVdVNRM="; + }; + + buildInputs = with pkgs; [ imlib2 xorg.libX11.dev ]; +} +``` + +### `buildInputs` and `nativeBuildInputs` +Running our favorite command again, yet more errors arise: + +```console +$ nix-build -E 'with import {}; callPackage ./icat.nix {}' +this derivation will be built: + /nix/store/0q6x7g7sz4pds3pgs8yb197fnf4r7rl2-icat.drv +building '/nix/store/0q6x7g7sz4pds3pgs8yb197fnf4r7rl2-icat.drv'... +unpacking sources +unpacking source archive /nix/store/y4750c9xljqy21b62a03z5xqvl3sd92q-source +source root is source +patching sources +configuring +no configure script, doing nothing +building +build flags: SHELL=/nix/store/7q1b1bsmxi91zci6g8714rcljl620y7f-bash-5.2-p15/bin/bash +gcc -c -Wall -pedantic -std=c99 -D_DEFAULT_SOURCE `pkg-config --cflags imlib2` -o icat.o icat.c +/nix/store/7q1b1bsmxi91zci6g8714rcljl620y7f-bash-5.2-p15/bin/bash: line 1: pkg-config: command not found +icat.c: In function 'main': +icat.c:319:33: warning: ignoring return value of 'write' declared with attribute 'warn_unused_result' [8;;https://gcc.gnu.org/onlinedocs/gcc/Warning-Options.html#index-Wunused-result-Wunused-result8;;] + 319 | write(tempfile, &buf, 1); + | ^~~~~~~~~~~~~~~~~~~~~~~~ +gcc -o icat icat.o `pkg-config --libs imlib2` +/nix/store/7q1b1bsmxi91zci6g8714rcljl620y7f-bash-5.2-p15/bin/bash: line 1: pkg-config: command not found +/nix/store/dx8hynidprz3kf4ngcjipnwaxp6h229f-binutils-2.40/bin/ld: icat.o: in function `resize_image_if_necessary': +icat.c:(.text+0x12a): undefined reference to `imlib_create_cropped_scaled_image' +/nix/store/dx8hynidprz3kf4ngcjipnwaxp6h229f-binutils-2.40/bin/ld: icat.c:(.text+0x132): undefined reference to `imlib_free_image_and_decache' +/nix/store/dx8hynidprz3kf4ngcjipnwaxp6h229f-binutils-2.40/bin/ld: icat.c:(.text+0x13a): undefined reference to `imlib_context_set_image' +/nix/store/dx8hynidprz3kf4ngcjipnwaxp6h229f-binutils-2.40/bin/ld: icat.o: in function `main': +icat.c:(.text.startup+0x25b): undefined reference to `imlib_load_image_immediately' +/nix/store/dx8hynidprz3kf4ngcjipnwaxp6h229f-binutils-2.40/bin/ld: icat.c:(.text.startup+0x26c): undefined reference to `imlib_context_set_image' +/nix/store/dx8hynidprz3kf4ngcjipnwaxp6h229f-binutils-2.40/bin/ld: icat.c:(.text.startup+0x271): undefined reference to `imlib_image_get_width' +/nix/store/dx8hynidprz3kf4ngcjipnwaxp6h229f-binutils-2.40/bin/ld: icat.c:(.text.startup+0x27a): undefined reference to `imlib_image_get_height' +/nix/store/dx8hynidprz3kf4ngcjipnwaxp6h229f-binutils-2.40/bin/ld: icat.c:(.text.startup+0x2ea): undefined reference to `imlib_image_query_pixel' +/nix/store/dx8hynidprz3kf4ngcjipnwaxp6h229f-binutils-2.40/bin/ld: icat.c:(.text.startup+0x2f8): undefined reference to `imlib_image_query_pixel' +/nix/store/dx8hynidprz3kf4ngcjipnwaxp6h229f-binutils-2.40/bin/ld: icat.c:(.text.startup+0x33c): undefined reference to `imlib_free_image_and_decache' +collect2: error: ld returned 1 exit status +make: *** [Makefile:23: icat] Error 1 +error: builder for '/nix/store/0q6x7g7sz4pds3pgs8yb197fnf4r7rl2-icat.drv' failed with exit code 2; + last 10 log lines: + > /nix/store/dx8hynidprz3kf4ngcjipnwaxp6h229f-binutils-2.40/bin/ld: icat.o: in function `main': + > icat.c:(.text.startup+0x25b): undefined reference to `imlib_load_image_immediately' + > /nix/store/dx8hynidprz3kf4ngcjipnwaxp6h229f-binutils-2.40/bin/ld: icat.c:(.text.startup+0x26c): undefined reference to `imlib_context_set_image' + > /nix/store/dx8hynidprz3kf4ngcjipnwaxp6h229f-binutils-2.40/bin/ld: icat.c:(.text.startup+0x271): undefined reference to `imlib_image_get_width' + > /nix/store/dx8hynidprz3kf4ngcjipnwaxp6h229f-binutils-2.40/bin/ld: icat.c:(.text.startup+0x27a): undefined reference to `imlib_image_get_height' + > /nix/store/dx8hynidprz3kf4ngcjipnwaxp6h229f-binutils-2.40/bin/ld: icat.c:(.text.startup+0x2ea): undefined reference to `imlib_image_query_pixel' + > /nix/store/dx8hynidprz3kf4ngcjipnwaxp6h229f-binutils-2.40/bin/ld: icat.c:(.text.startup+0x2f8): undefined reference to `imlib_image_query_pixel' + > /nix/store/dx8hynidprz3kf4ngcjipnwaxp6h229f-binutils-2.40/bin/ld: icat.c:(.text.startup+0x33c): undefined reference to `imlib_free_image_and_decache' + > collect2: error: ld returned 1 exit status + > make: *** [Makefile:23: icat] Error 1 + For full logs, run 'nix log /nix/store/0q6x7g7sz4pds3pgs8yb197fnf4r7rl2-icat.drv'. +``` + +There are several issues here, but the first one we can solve is `/nix/store/7q1b1bsmxi91zci6g8714rcljl620y7f-bash-5.2-p15/bin/bash: line 1: pkg-config: command not found`. According to the [Nixpkgs Manual](https://nixos.org/manual/nixpkgs/stable/#ssec-stdenv-dependencies), we should add dependencies to `buildInputs` if they're going to be copied or linked into the final output, or otherwise used somehow at runtime, but we should add dependencies to the `nativeBuildInputs` list if those dependencies are used at *build* time. `pkg-config` isn't needed after we build `icat`, so we'll add it to `nativeBuildInputs`: + +```nix +{ pkgs +, lib +, stdenv +}: + +stdenv.mkDerivation { + name = "icat"; + src = pkgs.fetchFromGitHub { + owner = "atextor"; + repo = "icat"; + rev = "master"; + sha256 = "sha256-b/2mRzCTyGkz2I1U+leUhspvW77VcHN7Awp+BVdVNRM="; + }; + + nativeBuildInputs = with pkgs; [ pkg-config ]; + buildInputs = with pkgs; [ imlib2 xorg.libX11.dev ]; +} +``` + +### Debugging with a Development Shell +This solves some of the errors we just saw, but not all; the `ld` error produced by all the undefined references is gone, but we still see a non-zero `make` return value: `make: *** No rule to make target 'install'`. + +Nix is automatically working with the `Makefile` that comes with `icat`, which indeed lacks an `install` target. The `README` in the `icat` repository only mentions using `make` to build the tool, presumably leaving installation up to us. We've now discovered one limit to what Nix can do for us automatically: it doesn't read minds. Fortunately, it does still make the fix quite straightforward to implement. + +If you haven't read the tutorials on creating [ad-hoc](https://nix.dev/tutorials/first-steps/dev-environment) or [declarative](https://nix.dev/tutorials/first-steps/declarative-and-reproducible-developer-environments) development environments, do that now before proceeding through the rest of this tutorial; dropping into a `nix-shell` is a crucial component in the Nix user toolbox, and indispensible for debugging. + +To enter a useful development shell, we'll pass the dependencies from `nativeBuildInputs` and `buildInputs` to `nix-shell -p`. We'll also make sure to include `git`, so we can clone the `icat` GitHub repository: + +```console +$ nix-shell -p pkg-config imlib2 xorg.libX11.dev git +``` + +After many lines of output about Nix copying dependencies, we can use the following commands to retrieve and build the `icat` source code: + +```console +$ git clone https://github.com/atextor/icat +$ cd icat +$ make +``` + +In the current `master` branch of `icat`, a warning is thrown when building: + +```console +[nix-shell:~/icat]$ make +gcc -c -Wall -pedantic -std=c99 -D_DEFAULT_SOURCE `pkg-config --cflags imlib2` -o icat.o icat.c +icat.c: In function ‘main’: +icat.c:319:33: warning: ignoring return value of ‘write’ declared with attribute ‘warn_unused_result’ [-Wunused-result] + 319 | write(tempfile, &buf, 1); + | ^~~~~~~~~~~~~~~~~~~~~~~~ +gcc -o icat icat.o `pkg-config --libs imlib2` +``` + +However, this does not prevent the binary from being produced; an `icat` executable is now present in the local directory, and it's up to us to decide what to do with it. + +### installPhase +In order to make packages available for other packages to depend on, Nix copies everything to the Nix store (at `/nix/store`), and symlinks them from there into build contexts and development environments. + +The `Makefile` doesn't provide an installation step, so we must produce one for our derivation, using the [`installPhase` attribute](https://nixos.org/manual/nixpkgs/stable/#ssec-install-phase), which contains a list of command strings to execute to accomplish the installation. + +The `icat` executable is only used at runtime, and isn't a compile-time input for anything else at this point, so we only need to concern ourselves with the `bin` output. In Nix, the result of a build is copied to a location stored in the `$out` variable accessible in the derivation's component scripts; we'll create a `bin` directory within that, and then copy our `icat` executable there: + +```nix +# icat.nix +{ pkgs +, lib +, stdenv +}: + +stdenv.mkDerivation { + name = "icat"; + src = pkgs.fetchFromGitHub { + owner = "atextor"; + repo = "icat"; + rev = "master"; + sha256 = "sha256-b/2mRzCTyGkz2I1U+leUhspvW77VcHN7Awp+BVdVNRM="; + }; + + nativeBuildInputs = with pkgs; [ pkg-config ]; + buildInputs = with pkgs; [ imlib2 xorg.libX11.dev ]; + + installPhase = '' + mkdir -p $out/bin + cp icat $out/bin + ''; +} +``` + +After running our `nix-build` command one last time, we can `ls` in the local directory to find a `result` symlink to the Nix store, with `result/bin/icat` the executable we built. Success! + +## Contributing our Work +Now that we've packaged `icat`, it's time to prepare it for submission upstream to Nixpkgs. + +### Building a Release Version +Our `icat.nix` definition uses the `master` revision of the upstream repository, which is suitable for individual use but not a great practice for submission to Nixpkgs; it would be better to use a fixed revision, corresponding to a particular release version of the software, at least so maintainers (perhaps you!) could easily check when this package should be updated. + +The upstream GitHub repository has [several tags available](https://github.com/atextor/icat/tags), which correspond to released versions. We'll modify our existing `icat.nix` to download and build the latest tag instead of what's available on `master`. This time, instead of using the `lib.fakeSha256` trick, we'll use [`nix-prefetch-url`](https://nixos.org/manual/nix/stable/command-ref/nix-prefetch-url.html) to retrieve the hash we need: + +```console +$ nix-prefetch-url --unpack https://github.com/atextor/icat/archive/refs/tags/v0.5.tar.gz +path is '/nix/store/p8jl1jlqxcsc7ryiazbpm7c1mqb6848b-v0.5.tar.gz' +0wyy2ksxp95vnh71ybj1bbmqd5ggp13x3mk37pzr99ljs9awy8ka +``` + +```nix +# icat.nix +{ pkgs +, lib +, stdenv +}: + +stdenv.mkDerivation { + name = "icat"; + src = pkgs.fetchFromGitHub { + owner = "atextor"; + repo = "icat"; + rev = "v0.5"; + sha256 = "0wyy2ksxp95vnh71ybj1bbmqd5ggp13x3mk37pzr99ljs9awy8ka"; + }; + + nativeBuildInputs = with pkgs; [ pkg-config ]; + buildInputs = with pkgs; [ imlib2 xorg.libX11.dev ]; + + installPhase = '' + mkdir -p $out/bin + cp icat $out/bin + ''; +} +``` + +And it builds! +```console +$ nix-build -E 'with import {}; callPackage ./icat.nix {}' +these 2 derivations will be built: + /nix/store/vvjyrngklzxbcsfiyp4hr1z2qcdqm8j7-source.drv + /nix/store/x6h1kfd4h16vhj0cxlakrm5igbbbz7v3-icat.drv +building '/nix/store/vvjyrngklzxbcsfiyp4hr1z2qcdqm8j7-source.drv'... + +trying https://github.com/atextor/icat/archive/v0.5.tar.gz + % Total % Received % Xferd Average Speed Time Time Time Current + Dload Upload Total Spent Left Speed + 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 +100 46232 0 46232 0 0 60947 0 --:--:-- --:--:-- --:--:-- 60947 +unpacking source archive /build/v0.5.tar.gz +building '/nix/store/x6h1kfd4h16vhj0cxlakrm5igbbbz7v3-icat.drv'... +unpacking sources +unpacking source archive /nix/store/rx21f6fgnmxgp1sw0wbqll9wds4xc6v0-source +source root is source +patching sources +configuring +no configure script, doing nothing +building +build flags: SHELL=/nix/store/7q1b1bsmxi91zci6g8714rcljl620y7f-bash-5.2-p15/bin/bash +gcc -c -Wall -pedantic -std=c99 -D_BSD_SOURCE -o icat.o icat.c +In file included from /nix/store/dpk5m64n0axk01fq8h2m0yl9hhpq2nqk-glibc-2.37-8-dev/include/bits/libc-header-start.h:33, + from /nix/store/dpk5m64n0axk01fq8h2m0yl9hhpq2nqk-glibc-2.37-8-dev/include/stdio.h:27, + from icat.c:31: +/nix/store/dpk5m64n0axk01fq8h2m0yl9hhpq2nqk-glibc-2.37-8-dev/include/features.h:195:3: warning: #warning "_BSD_SOURCE and _SVID_SOURCE are deprecated, use _DEFAULT_SOURCE" [8;;https://gcc.gnu.org/onlinedocs/gcc/Warning-Options.html#index-Wcpp-Wcpp8;;] + 195 | # warning "_BSD_SOURCE and _SVID_SOURCE are deprecated, use _DEFAULT_SOURCE" + | ^~~~~~~ +icat.c: In function 'main': +icat.c:319:33: warning: ignoring return value of 'write' declared with attribute 'warn_unused_result' [8;;https://gcc.gnu.org/onlinedocs/gcc/Warning-Options.html#index-Wunused-result-Wunused-result8;;] + 319 | write(tempfile, &buf, 1); + | ^~~~~~~~~~~~~~~~~~~~~~~~ +gcc -o icat icat.o -lImlib2 +installing +post-installation fixup +shrinking RPATHs of ELF executables and libraries in /nix/store/g6w508vxwr3df25dnl4k3xvcr4pqxprj-icat +shrinking /nix/store/g6w508vxwr3df25dnl4k3xvcr4pqxprj-icat/bin/icat +checking for references to /build/ in /nix/store/g6w508vxwr3df25dnl4k3xvcr4pqxprj-icat... +patching script interpreter paths in /nix/store/g6w508vxwr3df25dnl4k3xvcr4pqxprj-icat +stripping (with command strip and flags -S -p) in /nix/store/g6w508vxwr3df25dnl4k3xvcr4pqxprj-icat/bin +/nix/store/g6w508vxwr3df25dnl4k3xvcr4pqxprj-icat +``` + +We still see the unused-result warning thrown by the compiler, but the package successfully built, and the very last line of output tells us where Nix put the result. + +### Phases and Hooks +Nix package derivations are separated into [phases](https://nixos.org/manual/nixpkgs/unstable/#sec-stdenv-phases), each of which is intended to control some aspect of the build process. + +During derivation realisation, there are a number of shell functions ("hooks", in `nixpkgs`) which may execute in each derivation phase, which do things like set variables, source files, create directories, and so on. These are run both before and after each phase, controlling the build environment and helping to prevent environment-modifying behavior defined within packages from creating sources of nondeterminism within and between Nix derivations. + +It's good practice when packaging for `nixpkgs` to include calls to these hooks in the derivation phases you define, even when you don't make direct use of them; this facilitates easy [overriding](https://nixos.org/manual/nixpkgs/stable/#chap-overrides) of specific parts of the derivation later, in addition to the previously-mentioned reproducibility benefits. + +Nix automatically determined the `buildPhase` information for our `icat` package, but we needed to define a custom `installPhase` which we should now adjust to call the appropriate hooks: + +```nix +# icat.nix +... + installPhase = '' + runHook preInstall + mkdir -p $out/bin + cp icat $out/bin + runHook postInstall + ''; +... +``` + +### Package Metadata +By convention, all packages in Nixpkgs have a `meta` attribute in their derivation, which contains information like a description of the package, the homepage of the project it belongs to, the software license, the platforms the package can be built for, and a list of Nixpkgs maintainers for the package. In this case, I'm the contributing user, so I'll add myself to the maintainers list for this package. + +:::{note} +Before contributing your first package, you must add your information to `nixpkgs/maintainers/maintainers-list.nix`, following the instructions [here](https://nixos.org/manual/nixpkgs/stable/#var-meta-maintainers). +::: + +Before we contribute our package, we should add this metadata to the `meta` attribute passed to `mkDerivation`, following the [contribution guidelines](https://nixos.org/manual/nixpkgs/stable/#reviewing-contributions-new-packages): + +```nix +# icat.nix +... + meta = with lib; { + description = "icat (Image cat) outputs images in 256-color capable terminals."; + homepage = "https://github.com/atextor/icat"; + license = licenses.bsdOriginal; + platforms = platforms.unix; + maintainers = [ maintainers.proofconstruction ]; + }; +... +``` + +### Adding to Nixpkgs +Now that our derivation is complete, we can copy it into an appropriate location within the `nixpkgs` source tree. + +`icat` is a console tool which also displays graphics, so there are a few places within Nixpkgs that we could reasonably move it. Presently, there are two other similar tools already in `nixpkgs`, both of which are stored in `pkgs/applications/graphics`, so we'll put `icat` in its own directory there too. + +We'll also keep with the convention of package derivation files being named `default.nix`: + +```console +$ git clone https://github.com/nixos/nixpkgs +$ mkdir nixpkgs/pkgs/applications/graphics/icat +$ mv icat.nix nixpkgs/pkgs/applications/graphics/icat/default.nix +``` + +The relative directory paths to all packages get referenced in `pkgs/top-level/all-packages.nix`, which we should update now, putting our new package next to the other similar tools: + +```nix +... +icat = callPackage ../applications/graphics/icat { }; +... +``` + +### One Final Test, One More Command +There are [several additional steps](https://nixos.org/manual/nixpkgs/stable/#submitting-changes-making-patches) we should perform before introducing a new package into Nixpkgs, but here we'll just make sure our package builds within its new context. For this, we'll use a new `nix-build` invocation, which we'll run from the root of the Nixpkgs repository: + +```console +nix-build -A icat + ^ ^ + 1 2 +``` + +This expression (1) selects the attribute corresponding to (2) the top-level expression to evaluate, which in this case is our new package `icat`. + +Let's change directories to the root of the Nixpkgs repo, and run it now: + +```console +$ cd nixpkgs +$ nix-build -A icat +this derivation will be built: + /nix/store/kf7na78dhwhfrgxjaj27gjkqs4b3n7wz-icat.drv +... + +... +/nix/store/vr2vk8z8839l5j6gra0qlyxrh5sarmh4-icat +``` + +The build completed as expected, and produced a `result` in our current directory. Our package is now ready for submission to Nixpkgs! From 0c2c29e27d306db8f760308519dccbcb51ca76c7 Mon Sep 17 00:00:00 2001 From: alex Date: Thu, 20 Jul 2023 11:43:39 -0500 Subject: [PATCH 02/74] mention that the stdenv automates a lot of this --- source/tutorials/packaging-existing-software.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/source/tutorials/packaging-existing-software.md b/source/tutorials/packaging-existing-software.md index 0f68d1dc7..dcaf93d4c 100644 --- a/source/tutorials/packaging-existing-software.md +++ b/source/tutorials/packaging-existing-software.md @@ -17,7 +17,7 @@ In the long term, Nix helps tremendously in alleviating that stress, but when *f If you haven't already read the tutorial on making a derivation, please go do so before reading this! -In this tutorial, we'll see how to create Nix derivations to package C/C++ software. +In this tutorial, we'll see how to create Nix derivations to package C/C++ software, taking advantage of the [`nixpkgs` `stdenv`](https://nixos.org/manual/nixpkgs/stable/#chap-stdenv) which automates much of the work of building self-contained C/C++ packages. We'll begin by considering `hello`, a feature-complete implementation of the famous "hello world", which requires no external dependencies. From 0a211946de840a976ce794fc43a6235b8230cb13 Mon Sep 17 00:00:00 2001 From: alex Date: Thu, 20 Jul 2023 11:48:26 -0500 Subject: [PATCH 03/74] remove contributing to nixpkgs section --- .../tutorials/packaging-existing-software.md | 43 ------------------- 1 file changed, 43 deletions(-) diff --git a/source/tutorials/packaging-existing-software.md b/source/tutorials/packaging-existing-software.md index dcaf93d4c..2e4afc74f 100644 --- a/source/tutorials/packaging-existing-software.md +++ b/source/tutorials/packaging-existing-software.md @@ -25,7 +25,6 @@ Then we'll move to progressively more complex packages with their own separate d Along the way, we'll encounter and address Nix error messages, build failures, and a host of other issues, developing our iterative debugging techniques as we go. -Finally, we'll wrap up by preparing these projects for contribution to `nixpkgs`. ## A Simple Project To start, we'll write a skeleton derivation, updating this as we go: @@ -822,49 +821,7 @@ Before we contribute our package, we should add this metadata to the `meta` attr ... ``` -### Adding to Nixpkgs -Now that our derivation is complete, we can copy it into an appropriate location within the `nixpkgs` source tree. - -`icat` is a console tool which also displays graphics, so there are a few places within Nixpkgs that we could reasonably move it. Presently, there are two other similar tools already in `nixpkgs`, both of which are stored in `pkgs/applications/graphics`, so we'll put `icat` in its own directory there too. - -We'll also keep with the convention of package derivation files being named `default.nix`: - -```console -$ git clone https://github.com/nixos/nixpkgs -$ mkdir nixpkgs/pkgs/applications/graphics/icat -$ mv icat.nix nixpkgs/pkgs/applications/graphics/icat/default.nix -``` - -The relative directory paths to all packages get referenced in `pkgs/top-level/all-packages.nix`, which we should update now, putting our new package next to the other similar tools: - -```nix -... -icat = callPackage ../applications/graphics/icat { }; -... -``` - -### One Final Test, One More Command -There are [several additional steps](https://nixos.org/manual/nixpkgs/stable/#submitting-changes-making-patches) we should perform before introducing a new package into Nixpkgs, but here we'll just make sure our package builds within its new context. For this, we'll use a new `nix-build` invocation, which we'll run from the root of the Nixpkgs repository: - -```console -nix-build -A icat - ^ ^ - 1 2 -``` - -This expression (1) selects the attribute corresponding to (2) the top-level expression to evaluate, which in this case is our new package `icat`. - -Let's change directories to the root of the Nixpkgs repo, and run it now: ```console -$ cd nixpkgs -$ nix-build -A icat -this derivation will be built: - /nix/store/kf7na78dhwhfrgxjaj27gjkqs4b3n7wz-icat.drv -... - -... -/nix/store/vr2vk8z8839l5j6gra0qlyxrh5sarmh4-icat ``` -The build completed as expected, and produced a `result` in our current directory. Our package is now ready for submission to Nixpkgs! From ce6bd9a9130a5a610fea01918b49c56ac491a176 Mon Sep 17 00:00:00 2001 From: alex Date: Thu, 20 Jul 2023 11:49:03 -0500 Subject: [PATCH 04/74] add explanation of what "package" means here --- source/tutorials/packaging-existing-software.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/source/tutorials/packaging-existing-software.md b/source/tutorials/packaging-existing-software.md index 2e4afc74f..ad00e6b27 100644 --- a/source/tutorials/packaging-existing-software.md +++ b/source/tutorials/packaging-existing-software.md @@ -25,6 +25,10 @@ Then we'll move to progressively more complex packages with their own separate d Along the way, we'll encounter and address Nix error messages, build failures, and a host of other issues, developing our iterative debugging techniques as we go. +## Packages in Nix +Before we proceed, an important point of clarification: we conventionally use the term "package" by analogy to other systems, although this term is not a proper concept in Nix. + +For the purposes of this tutorial, by "package" we mean "a Nix function which takes an attribute set of 'dependencies' and produces a derivation", where "dependencies" could be other packages or configuration parameters. ## A Simple Project To start, we'll write a skeleton derivation, updating this as we go: From 77daebd1273aa3a9550cceb9e13c411e81c51a09 Mon Sep 17 00:00:00 2001 From: alex Date: Thu, 20 Jul 2023 11:49:42 -0500 Subject: [PATCH 05/74] rephrase fetchTarball introduction --- source/tutorials/packaging-existing-software.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/source/tutorials/packaging-existing-software.md b/source/tutorials/packaging-existing-software.md index ad00e6b27..f067c7708 100644 --- a/source/tutorials/packaging-existing-software.md +++ b/source/tutorials/packaging-existing-software.md @@ -43,9 +43,9 @@ Since GNU Hello is a popular (in a certain sense) package from the GNU Project, In this case, we'll download the [latest version](https://ftp.gnu.org/gnu/hello/hello-2.12.1.tar.gz) of `hello`, currently `2.12.1`. -Downloading that into our build context is a good first step; we can do this in several ways, but it's best to use one of the nixpkgs builtin fetcher functions, in this case `fetchTarball` from nixpkgs. +Downloading that into our build context is a good first step; we can do this in several ways, but it's best to use one of the nixpkgs builtin fetcher functions. -`fetchTarball` takes as argument an attribute set containing the URI path to the download and a SHA256 hash of the file. +In this case, we'll use `fetchTarball`, which takes the URI path to the download file and a SHA256 hash of its contents. We won't know what the hash really is until we download it, but fortunately Nix will complain at us if we're wrong, so we can fake it with a bunch of zeroes (53 to be exact): From bbb4a130b634f687db944ba8c307ee11a45d04d8 Mon Sep 17 00:00:00 2001 From: alex Date: Thu, 20 Jul 2023 11:56:09 -0500 Subject: [PATCH 06/74] use lib.fakeSha256 instead of a string of 53 zeroes --- source/tutorials/packaging-existing-software.md | 14 ++++---------- 1 file changed, 4 insertions(+), 10 deletions(-) diff --git a/source/tutorials/packaging-existing-software.md b/source/tutorials/packaging-existing-software.md index f067c7708..78438ced5 100644 --- a/source/tutorials/packaging-existing-software.md +++ b/source/tutorials/packaging-existing-software.md @@ -47,7 +47,7 @@ Downloading that into our build context is a good first step; we can do this in In this case, we'll use `fetchTarball`, which takes the URI path to the download file and a SHA256 hash of its contents. -We won't know what the hash really is until we download it, but fortunately Nix will complain at us if we're wrong, so we can fake it with a bunch of zeroes (53 to be exact): +Here is our first iterative debugging technique: we can't actually know the hash until after we've downloaded and unpacked the tarball, but Nix will complain at us if the hash we supplied was incorrect, so we can just supply a fake one with `lib.fakeSha256` and change our derivation after Nix informs us of the correct hash: ```nix # hello.nix @@ -55,15 +55,11 @@ We won't know what the hash really is until we download it, but fortunately Nix stdenv.mkDerivation { src = builtins.fetchTarball { url = "https://ftp.gnu.org/gnu/hello/hello-2.12.1.tar.gz"; - sha256 = "0000000000000000000000000000000000000000000000000000"; + sha256 = lib.fakeSha256; }; } ``` -:::{note} -Instead of typing 53 zeroes, you can use `lib.fakeSha256`. We'll do this in later examples. -::: - Let's save this file to `hello.nix` and try to build it. To do so, we'll use `nix-build`... ```console @@ -118,7 +114,7 @@ stdenv.mkDerivation { name = "hello"; src = builtins.fetchTarball { url = "https://ftp.gnu.org/gnu/hello/hello-2.12.1.tar.gz"; - sha256 = "0000000000000000000000000000000000000000000000000000"; + sha256 = lib.fakeSha256; }; } ``` @@ -149,7 +145,7 @@ error: ``` ### Finding The File Hash -As expected, Nix complained at us for lying about the file hash, and helpfully provided the correct one. We can substitute this into our `hello.nix` file, replacing the string of zeroes: +As expected, Nix complained at us for lying about the file hash, and helpfully provided the correct one. We can substitute this into our `hello.nix` file, replacing `lib.fakeSha256`: ```nix # hello.nix @@ -265,7 +261,6 @@ Rather than `url` and `sha256`, we must now supply the following arguments: - `owner`: a string representing the user or organization account which owns the repository. The `nixpkgs` repository lives at `https://github.com/NixOS/nixpkgs`, so if we were fetching the `nixpkgs` source, for example, and in this case the source is hosted at `https://github.com/atextor/icat`, so we want `owner = "atextor"`. - `repo`: another string corresponding to the name of the repository. Here we'll use `repo = "icat"`. - `rev`: the *revision*, such as the git commit hash or tag (e.g. `v1.0`), to download from GitHub. On GitHub, you can find these on the [Releases page](https://github.com/atextor/icat/releases), but for now we're just going to use `master`. -- `hash`: similar to the earlier `hello` example, this corresponds to the hash of the extracted directory after it is downloaded. Several hash algorithms are available, but we used SHA256 before, so we'll do it again here and use `sha256` in place of the `hash` input. We could fake the hash with 53 zeroes (or any string of characters that doesn't match the *actual* hash) as we did before, but rather than type that out, we'll just use `lib.fakeSha256`. Updating our file accordingly: @@ -388,7 +383,6 @@ stdenv.mkDerivation { } ``` -We used `lib.fakeSha256`, which does essentially the same thing as our earlier 53 zeroes trick: ```console $ nix-build -E 'with import {}; callPackage ./icat.nix {}' From f1d6bf20bd8f887bece23ec046007d360eac3fdc Mon Sep 17 00:00:00 2001 From: alex Date: Thu, 20 Jul 2023 12:01:39 -0500 Subject: [PATCH 07/74] note what callPackage does --- source/tutorials/packaging-existing-software.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/source/tutorials/packaging-existing-software.md b/source/tutorials/packaging-existing-software.md index 78438ced5..72668a31a 100644 --- a/source/tutorials/packaging-existing-software.md +++ b/source/tutorials/packaging-existing-software.md @@ -91,7 +91,7 @@ with import {}; callPackage ./hello.nix {} 1 2 3 4 5 6 7 ``` -This expression imports (2) the system `nixpkgs` (3) into the local scope using the `with` statement (1), applying it (remember, like any other derivation in Nix, `nixpkgs` is actually a function!) to an empty attribute set (4). +`callPackage` automatically passes attributes from `pkgs` to the given function, if they match attributes required by that function's argument attrset. Here, `callPackage` will supply `pkgs`, `lib`, and `stdenv`. From `nixpkgs`, we use the `callPackage` function (5) to import our `hello.nix` (6). Since the `hello.nix` derivation [is a function](https://nixos.org/manual/nix/stable/language/derivations.html) which takes two arguments [it will already be passed](), we also apply it to an empty attribute set (7). Although `hello.nix` takes a `pkgs` argument, we don't need to apply the function to `nixpkgs`, because the `with` statement has already included the `nixpkgs` we just imported into the local context. From e88e060e677a3989e4c6e663cdbb04c11d7bab55 Mon Sep 17 00:00:00 2001 From: alex Date: Thu, 20 Jul 2023 12:04:36 -0500 Subject: [PATCH 08/74] let the manual more-verbosely explain fetchFromGitHub arguments --- source/tutorials/packaging-existing-software.md | 14 +++++++++----- 1 file changed, 9 insertions(+), 5 deletions(-) diff --git a/source/tutorials/packaging-existing-software.md b/source/tutorials/packaging-existing-software.md index 72668a31a..f63f82ad7 100644 --- a/source/tutorials/packaging-existing-software.md +++ b/source/tutorials/packaging-existing-software.md @@ -256,11 +256,6 @@ stdenv.mkDerivation { } ``` -### Fetching Source from GitHub -Rather than `url` and `sha256`, we must now supply the following arguments: -- `owner`: a string representing the user or organization account which owns the repository. The `nixpkgs` repository lives at `https://github.com/NixOS/nixpkgs`, so if we were fetching the `nixpkgs` source, for example, and in this case the source is hosted at `https://github.com/atextor/icat`, so we want `owner = "atextor"`. -- `repo`: another string corresponding to the name of the repository. Here we'll use `repo = "icat"`. -- `rev`: the *revision*, such as the git commit hash or tag (e.g. `v1.0`), to download from GitHub. On GitHub, you can find these on the [Releases page](https://github.com/atextor/icat/releases), but for now we're just going to use `master`. Updating our file accordingly: @@ -383,6 +378,15 @@ stdenv.mkDerivation { } ``` +### Fetching Source from GitHub +While `fetchTarball` required `url` and `sha256` arguments, we'll need more than that for [`fetchFromGitHub`](https://nixos.org/manual/nixpkgs/stable/#fetchfromgithub). + +The source we want is hosted on GitHub at `https://github.com/atextor/icat`, which already gives us the first two arguments: +- `owner`: the name of the account controlling the repository; `owner = "atextor"` +- `repo`: the name of the repository we want to fetch; `repo = "icat"` + +We can navigate to the project's [Releases page](https://github.com/atextor/icat/releases) to find a suitable `rev`, such as the git commit hash or tag (e.g. `v1.0`) corresponding to the release we want to fetch. In this case, the latest release tag is `v0.5`. + ```console $ nix-build -E 'with import {}; callPackage ./icat.nix {}' From a1eea0bf779ea39c63081ffcb06f9374943fc92f Mon Sep 17 00:00:00 2001 From: alex Date: Thu, 20 Jul 2023 12:08:28 -0500 Subject: [PATCH 09/74] the stdenv is doing the work, not nix itself --- source/tutorials/packaging-existing-software.md | 6 +----- 1 file changed, 1 insertion(+), 5 deletions(-) diff --git a/source/tutorials/packaging-existing-software.md b/source/tutorials/packaging-existing-software.md index f63f82ad7..02bcb80a3 100644 --- a/source/tutorials/packaging-existing-software.md +++ b/source/tutorials/packaging-existing-software.md @@ -182,11 +182,7 @@ build flags: SHELL=/nix/store/7q1b1bsmxi91zci6g8714rcljl620y7f-bash-5.2-p15/bin/ ``` Great news: our derivation built successfully! -:::{important} -But how? This looks magical! -::: - -We can see from the console output that `configure` was called, which produced a `Makefile` that was then used to build the project; we didn't actually write any build instructions, so we can surmise that Nix automatically detected the structure of the project directory. Indeed, the build system in Nix is based on `autoconf`. +We can see from the console output that `configure` was called, which produced a `Makefile` that was then used to build the project; we didn't actually write any build instructions, so we can surmise that Nix automatically detected the structure of the project directory. Indeed, the build system in `stdenv` is based on `autoconf`. ### Build Result We can check our working directory for the result: From 0797d1cd4e541ad96eb65f0fc0fcc63357a268a9 Mon Sep 17 00:00:00 2001 From: alex Date: Thu, 20 Jul 2023 13:29:26 -0500 Subject: [PATCH 10/74] remove namespace errors --- .../tutorials/packaging-existing-software.md | 132 +----------------- 1 file changed, 2 insertions(+), 130 deletions(-) diff --git a/source/tutorials/packaging-existing-software.md b/source/tutorials/packaging-existing-software.md index 02bcb80a3..433c897a6 100644 --- a/source/tutorials/packaging-existing-software.md +++ b/source/tutorials/packaging-existing-software.md @@ -206,106 +206,7 @@ The `hello` program is a simple and common place to start packaging, but it's no Now, we'll look at packaging a somewhat more complicated program, `icat`, which allows us to render images in our terminal. -Though there are at least two alternative similar tools already in `nixpkgs`, at the time of writing, this particular tool hasn't been packaged, and it is used in {ref}`another tutorial `.) *fixme: complete and merge https://github.com/NixOS/nix.dev/pull/645, then link it here*, so this is a good opportunity to do something both informative and useful. - -We'll start by copying the `hello.nix` from the previous section to a new file, `icat.nix`: - -```nix -# icat.nix -{ pkgs, stdenv }: -stdenv.mkDerivation { - name = "hello"; - src = builtins.fetchTarball { - url = "https://ftp.gnu.org/gnu/hello/hello-2.12.1.tar.gz"; - sha256 = "0xw6cr5jgi1ir13q6apvrivwmmpr5j8vbymp0x6ll0kcv6366hnn"; - }; -} -``` - -While Nix can sometimes feel magic, it's not *actually* magic, so unfortunately this won't magically produce `icat` for us, and we'll need to make several changes. - -To start, we'll need to change the `name` attribute: - -```nix -# icat.nix -{ pkgs, stdenv }: -stdenv.mkDerivation { - name = "icat"; - src = builtins.fetchTarball { - url = "https://ftp.gnu.org/gnu/hello/hello-2.12.1.tar.gz"; - sha256 = "0xw6cr5jgi1ir13q6apvrivwmmpr5j8vbymp0x6ll0kcv6366hnn"; - }; -} -``` - -Now we'll download the source code. `icat`'s upstream repository is hosted on [GitHub](https://github.com/atextor/icat), so we should slightly modify our previous [source fetcher](https://nixos.org/manual/nixpkgs/stable/#chap-pkgs-fetchers): instead of `fetchTarball`, we'll use [`fetchFromGitHub`](https://nixos.org/manual/nixpkgs/stable/#fetchfromgithub): - -```nix -# icat.nix -{ pkgs, stdenv }: -stdenv.mkDerivation { - name = "icat"; - src = builtins.fetchFromGitHub { - url = "https://ftp.gnu.org/gnu/hello/hello-2.12.1.tar.gz"; - sha256 = "0xw6cr5jgi1ir13q6apvrivwmmpr5j8vbymp0x6ll0kcv6366hnn"; - }; -} -``` - - -Updating our file accordingly: - -```nix -# icat.nix -{ pkgs, stdenv }: -stdenv.mkDerivation { - name = "icat"; - src = builtins.fetchFromGitHub { - owner = "atextor"; - repo = "icat"; - rev = "master"; - sha256 = lib.fakeSha256; - }; -} -``` - -Running our previous `nix-build` invocation: - -```console -$ nix-build -E 'with import {}; callPackage ./icat.nix {}' -error: - … while evaluating a branch condition - - at /nix/store/i6w7hmdjp1jg71g7xbjgz5rn96q443c6-nixos-23.05.1471.b72aa95f7f0/nixos/lib/customisation.nix:179:8: - - 178| - 179| in if missingArgs == [] then makeOverridable f allArgs else abort error; - | ^ - 180| - - … while calling the 'attrNames' builtin - - at /nix/store/i6w7hmdjp1jg71g7xbjgz5rn96q443c6-nixos-23.05.1471.b72aa95f7f0/nixos/lib/customisation.nix:139:21: - - 138| # wouldn't be passed to it - 139| missingArgs = lib.attrNames - | ^ - 140| # Filter out arguments that have a default value - - (stack trace truncated; use '--show-trace' to show the full trace) - - error: undefined variable 'lib' - - at /home/nix-user/icat.nix:9:12: - - 8| rev = "master"; - 9| hash = lib.fakeSha256; - | ^ - 10| }; -``` - -### Namespacing -This one is easy: `lib` lives in the `pkgs` namespace, so we can either fix this by invoking `pkgs.lib.fakeSha256` instead, or by taking `lib` as an argument to the whole expression. The latter option is more common, so we'll do that. This is also a good time to rearrange our set of arguments to better conform to the [Nixpkgs syntactic conventions](https://nixos.org/manual/nixpkgs/stable/#chap-conventions): +We'll start by copying `hello.nix` from the previous section to a new file, `icat.nix`. Then we'll make several changes, starting with the `name` attribute: ```nix # icat.nix @@ -325,36 +226,7 @@ stdenv.mkDerivation { } ``` -```console -$ nix-build -E 'with import {}; callPackage ./icat.nix {}' -error: - … while calling the 'derivationStrict' builtin - - at /builtin/derivation.nix:9:12: (source not available) - - … while evaluating derivation 'icat' - whose name attribute is located at /nix/store/i6w7hmdjp1jg71g7xbjgz5rn96q443c6-nixos-23.05.1471.b72aa95f7f0/nixos/pkgs/stdenv/generic/make-derivation.nix:303:7 - - … while evaluating attribute 'src' of derivation 'icat' - - at /home/nix-user/icat.nix:9:3: - - 8| name = "icat"; - 9| src = builtins.fetchFromGitHub { - | ^ - 10| owner = "atextor"; - - error: attribute 'fetchFromGitHub' missing - - at /home/nix-user/icat.nix:9:9: - - 8| name = "icat"; - 9| src = builtins.fetchFromGitHub { - | ^ - 10| owner = "atextor"; -``` - -Another issue, and the converse of the previous one: `fetchFromGitHub` doesn't live in `builtins`, it lives in `pkgs`: +Now we'll download the source code. `icat`'s upstream repository is hosted on [GitHub](https://github.com/atextor/icat), so we should slightly modify our previous [source fetcher](https://nixos.org/manual/nixpkgs/stable/#chap-pkgs-fetchers): instead of `builtins.fetchTarball`, we'll use `pkgs.fetchFromGitHub`: ```nix # icat.nix From c70df9511cf4e7df31e3e650fc94d2b3c649ff24 Mon Sep 17 00:00:00 2001 From: alex Date: Thu, 20 Jul 2023 13:37:46 -0500 Subject: [PATCH 11/74] move dependencies to derivation inputs, because we use callPackage --- .../tutorials/packaging-existing-software.md | 37 +++++++++++++++---- 1 file changed, 29 insertions(+), 8 deletions(-) diff --git a/source/tutorials/packaging-existing-software.md b/source/tutorials/packaging-existing-software.md index 433c897a6..7b368818c 100644 --- a/source/tutorials/packaging-existing-software.md +++ b/source/tutorials/packaging-existing-software.md @@ -34,7 +34,10 @@ For the purposes of this tutorial, by "package" we mean "a Nix function which ta To start, we'll write a skeleton derivation, updating this as we go: ```nix -{ pkgs, stdenv }: +{ pkgs +, stdenv +}: + stdenv.mkDerivation { }; ``` @@ -51,7 +54,11 @@ Here is our first iterative debugging technique: we can't actually know the hash ```nix # hello.nix -{ pkgs, stdenv }: +{ pkgs +, lib +, stdenv +}: + stdenv.mkDerivation { src = builtins.fetchTarball { url = "https://ftp.gnu.org/gnu/hello/hello-2.12.1.tar.gz"; @@ -73,9 +80,9 @@ error: cannot evaluate a function that has an argument without a value ('pkgs') at /home/nix-user/hello.nix:2:3: 1| # hello.nix - 2| { pkgs, stdenv }: + 2| { pkgs | ^ - 3| stdenv.mkDerivation { + 3| , lib ``` ... and immediately run into a problem: every derivation is a *function*, and functions need *arguments*! @@ -109,7 +116,11 @@ Every derivation needs a `name` attribute, which must either be set directly or Let's update the file again to add a `name`: ```nix -{ pkgs, stdenv }: +{ pkgs +, lib +, stdenv +}: + stdenv.mkDerivation { name = "hello"; src = builtins.fetchTarball { @@ -149,7 +160,11 @@ As expected, Nix complained at us for lying about the file hash, and helpfully p ```nix # hello.nix -{ pkgs, stdenv }: +{ pkgs +, lib +, stdenv +}: + stdenv.mkDerivation { name = "hello"; src = builtins.fetchTarball { @@ -334,13 +349,14 @@ We can add this package to our build environment by either - adding `imlib2` to the set of inputs to the expression in `icat.nix`, and then adding `imlib2` to the list of `buildInputs` in `stdenv.mkDerivation`, or - adding `pkgs.imlib2` to the `buildInputs` directly, since `pkgs` is already in-scope. -We'll do the latter of these here: +Because `callPackage` is used to provide all necessary inputs in `nixpkgs` as well as in our `nix-build` invocation, the first approach is the one currently favored, and we'll use it here: ```nix # icat.nix { pkgs , lib , stdenv +, imlib2 }: stdenv.mkDerivation { @@ -352,7 +368,7 @@ stdenv.mkDerivation { sha256 = "sha256-b/2mRzCTyGkz2I1U+leUhspvW77VcHN7Awp+BVdVNRM="; }; - buildInputs = [ pkgs.imlib2 ]; + buildInputs = [ imlib2 ]; } ``` @@ -398,9 +414,12 @@ error: builder for '/nix/store/0csqp747mfw0v9n103abxgx611s6dkxm-icat.drv' failed In Nixpkgs, `Xlib` lives in the `dev` output of `xorg.libX11`, which we can add to `buildInputs` again with `pkgs.xorg.libX11.dev`. To avoid repeating ourselves, we can add `pkgs` to the local scope in `buildInputs` by using the [`with` statement](https://nixos.org/guides/nix-pills/basics-of-language.html#idm140737320521984): ```nix +# icat.nix { pkgs , lib , stdenv +, imlib2 +, xorg }: stdenv.mkDerivation { @@ -538,6 +557,8 @@ The `icat` executable is only used at runtime, and isn't a compile-time input fo { pkgs , lib , stdenv +, imlib2 +, xorg }: stdenv.mkDerivation { From a4233f2d8b94eb81d33964563be3ec75f03665e6 Mon Sep 17 00:00:00 2001 From: alex Date: Thu, 20 Jul 2023 13:42:28 -0500 Subject: [PATCH 12/74] use latest release instead of master --- source/tutorials/packaging-existing-software.md | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/source/tutorials/packaging-existing-software.md b/source/tutorials/packaging-existing-software.md index 7b368818c..c38b413ab 100644 --- a/source/tutorials/packaging-existing-software.md +++ b/source/tutorials/packaging-existing-software.md @@ -298,9 +298,9 @@ stdenv.mkDerivation { name = "icat"; src = pkgs.fetchFromGitHub { owner = "atextor"; - repo = "icat"; - rev = "master"; - sha256 = "sha256-b/2mRzCTyGkz2I1U+leUhspvW77VcHN7Awp+BVdVNRM="; + repo = "icat"; + rev = "v0.5"; + sha256 = "0wyy2ksxp95vnh71ybj1bbmqd5ggp13x3mk37pzr99ljs9awy8ka"; }; } ``` @@ -363,9 +363,9 @@ stdenv.mkDerivation { name = "icat"; src = pkgs.fetchFromGitHub { owner = "atextor"; - repo = "icat"; - rev = "master"; - sha256 = "sha256-b/2mRzCTyGkz2I1U+leUhspvW77VcHN7Awp+BVdVNRM="; + repo = "icat"; + rev = "v0.5"; + sha256 = "0wyy2ksxp95vnh71ybj1bbmqd5ggp13x3mk37pzr99ljs9awy8ka"; }; buildInputs = [ imlib2 ]; @@ -427,8 +427,8 @@ stdenv.mkDerivation { src = pkgs.fetchFromGitHub { owner = "atextor"; repo = "icat"; - rev = "master"; - sha256 = "sha256-b/2mRzCTyGkz2I1U+leUhspvW77VcHN7Awp+BVdVNRM="; + rev = "v0.5"; + sha256 = "0wyy2ksxp95vnh71ybj1bbmqd5ggp13x3mk37pzr99ljs9awy8ka"; }; buildInputs = with pkgs; [ imlib2 xorg.libX11.dev ]; From 42562620f3b9c61df6e65871cd67db037e1e448c Mon Sep 17 00:00:00 2001 From: alex Date: Thu, 20 Jul 2023 13:56:52 -0500 Subject: [PATCH 13/74] use nix-prefetch-url earlier --- source/tutorials/packaging-existing-software.md | 17 +++++------------ 1 file changed, 5 insertions(+), 12 deletions(-) diff --git a/source/tutorials/packaging-existing-software.md b/source/tutorials/packaging-existing-software.md index c38b413ab..f60c15ce1 100644 --- a/source/tutorials/packaging-existing-software.md +++ b/source/tutorials/packaging-existing-software.md @@ -270,22 +270,15 @@ The source we want is hosted on GitHub at `https://github.com/atextor/icat`, whi We can navigate to the project's [Releases page](https://github.com/atextor/icat/releases) to find a suitable `rev`, such as the git commit hash or tag (e.g. `v1.0`) corresponding to the release we want to fetch. In this case, the latest release tag is `v0.5`. +As in the `hello` example, we also need to supply a hash. This time, instead of using `lib.fakeSha256` and letting `nix-build` report the correct one in an error, we'll fetch the correct hash in the first place with the `nix-prefetch-url` command. We want the SHA256 hash of the *contents* of the tarball, so we need to pass the `--unpack` and `--type sha256` arguments too: ```console -$ nix-build -E 'with import {}; callPackage ./icat.nix {}' -... -unpacking source archive /build/master.tar.gz -error: hash mismatch in fixed-output derivation '/nix/store/lgjf8cq63ahqnd3b117g1q58g4nkprmj-source.drv': - specified: sha256-AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA= - got: sha256-b/2mRzCTyGkz2I1U+leUhspvW77VcHN7Awp+BVdVNRM= -error: 1 dependencies of derivation '/nix/store/afiw4a1l04pi82k6w630d42iflgfxbl6-icat.drv' failed to build +$ nix-prefetch-url --unpack https://github.com/atextor/icat/archive/refs/tags/v0.5.tar.gz --type sha256 +path is '/nix/store/p8jl1jlqxcsc7ryiazbpm7c1mqb6848b-v0.5.tar.gz' +0wyy2ksxp95vnh71ybj1bbmqd5ggp13x3mk37pzr99ljs9awy8ka ``` -:::{note} -We've been faking the hash and letting `nix-build` report the correct one in an error, but we could also fetch the correct hash in the first place with one of the `nix-prefetch` commands, or by downloading the tarball and passing it to the appropriate `nix-hash` invocation. We'll use the `nix-prefetch` trick in the next section. -::: - -Now that we have the correct hash, we'll replace `lib.fakeSha256` in the file and re-run the command: +Now we can supply the correct hash to `fetchFromGitHub`: ```nix # icat.nix From 5f6280bb3994b39e2ea5718643a2d673b2727994 Mon Sep 17 00:00:00 2001 From: alex Date: Thu, 20 Jul 2023 14:04:21 -0500 Subject: [PATCH 14/74] different source means different console output --- .../tutorials/packaging-existing-software.md | 133 +++++++++--------- 1 file changed, 63 insertions(+), 70 deletions(-) diff --git a/source/tutorials/packaging-existing-software.md b/source/tutorials/packaging-existing-software.md index f60c15ce1..229333cbc 100644 --- a/source/tutorials/packaging-existing-software.md +++ b/source/tutorials/packaging-existing-software.md @@ -304,36 +304,41 @@ Now we run into an entirely new issue: ```console $ nix-build -E 'with import {}; callPackage ./icat.nix {}' this derivation will be built: - /nix/store/dvqbkap31salw9mbr2xhxnnnb0089x3v-icat.drv -building '/nix/store/dvqbkap31salw9mbr2xhxnnnb0089x3v-icat.drv'... + /nix/store/al2wld63c66p3ln0rxqlkqqrqpspnicj-icat.drv +building '/nix/store/al2wld63c66p3ln0rxqlkqqrqpspnicj-icat.drv'... unpacking sources -unpacking source archive /nix/store/y4750c9xljqy21b62a03z5xqvl3sd92q-source +unpacking source archive /nix/store/rx21f6fgnmxgp1sw0wbqll9wds4xc6v0-source source root is source patching sources configuring no configure script, doing nothing building build flags: SHELL=/nix/store/7q1b1bsmxi91zci6g8714rcljl620y7f-bash-5.2-p15/bin/bash -gcc -c -Wall -pedantic -std=c99 -D_DEFAULT_SOURCE `pkg-config --cflags imlib2` -o icat.o icat.c -/nix/store/7q1b1bsmxi91zci6g8714rcljl620y7f-bash-5.2-p15/bin/bash: line 1: pkg-config: command not found +gcc -c -Wall -pedantic -std=c99 -D_BSD_SOURCE -o icat.o icat.c +In file included from /nix/store/dpk5m64n0axk01fq8h2m0yl9hhpq2nqk-glibc-2.37-8-dev/include/bits/libc-header-start.h:33, + from /nix/store/dpk5m64n0axk01fq8h2m0yl9hhpq2nqk-glibc-2.37-8-dev/include/stdio.h:27, + from icat.c:31: +/nix/store/dpk5m64n0axk01fq8h2m0yl9hhpq2nqk-glibc-2.37-8-dev/include/features.h:195:3: warning: #warning "_BSD_SOURCE and _SVID_SOURCE are deprecated, use _DEFAULT_SOURCE" [8;;https://gcc.gnu.org/onlinedocs/gcc/Warning-Options.html#index-Wcpp-Wcpp8;;] + 195 | # warning "_BSD_SOURCE and _SVID_SOURCE are deprecated, use _DEFAULT_SOURCE" + | ^~~~~~~ icat.c:39:10: fatal error: Imlib2.h: No such file or directory 39 | #include | ^~~~~~~~~~ compilation terminated. -make: *** [Makefile:20: icat.o] Error 1 -error: builder for '/nix/store/dvqbkap31salw9mbr2xhxnnnb0089x3v-icat.drv' failed with exit code 2; +make: *** [Makefile:16: icat.o] Error 1 +error: builder for '/nix/store/al2wld63c66p3ln0rxqlkqqrqpspnicj-icat.drv' failed with exit code 2; last 10 log lines: - > no configure script, doing nothing - > building - > build flags: SHELL=/nix/store/7q1b1bsmxi91zci6g8714rcljl620y7f-bash-5.2-p15/bin/bash - > gcc -c -Wall -pedantic -std=c99 -D_DEFAULT_SOURCE `pkg-config --cflags imlib2` -o icat.o icat.c - > /nix/store/7q1b1bsmxi91zci6g8714rcljl620y7f-bash-5.2-p15/bin/bash: line 1: pkg-config: command not found + > from /nix/store/dpk5m64n0axk01fq8h2m0yl9hhpq2nqk-glibc-2.37-8-dev/include/stdio.h:27, + > from icat.c:31: + > /nix/store/dpk5m64n0axk01fq8h2m0yl9hhpq2nqk-glibc-2.37-8-dev/include/features.h:195:3: warning: #warning "_BSD_SOURCE and _SVID_SOURCE are deprecated, use _DEFAULT_SOURCE" [8;;https://gcc.gnu.org/onlinedocs/gcc/Warning-Options.html#index-Wcpp-Wcpp8;;] + > 195 | # warning "_BSD_SOURCE and _SVID_SOURCE are deprecated, use _DEFAULT_SOURCE" + > | ^~~~~~~ > icat.c:39:10: fatal error: Imlib2.h: No such file or directory > 39 | #include > | ^~~~~~~~~~ > compilation terminated. - > make: *** [Makefile:20: icat.o] Error 1 - For full logs, run 'nix log /nix/store/dvqbkap31salw9mbr2xhxnnnb0089x3v-icat.drv'. + > make: *** [Makefile:16: icat.o] Error 1 + For full logs, run 'nix log /nix/store/al2wld63c66p3ln0rxqlkqqrqpspnicj-icat.drv'. ``` Finally, a compiler error! We've successfully pulled the `icat` source from GitHub, and Nix tried to build what it found, but is missing a dependency: the `imlib2` header. If we [search for `imlib2` on search.nixos.org](https://search.nixos.org/packages?channel=23.05&from=0&size=50&sort=relevance&type=packages&query=imlib2), we'll find that `imlib2` is already in `nixpkgs`. @@ -371,37 +376,42 @@ Another error, but we get further this time: ```console $ nix-build -E 'with import {}; callPackage ./icat.nix {}' this derivation will be built: - /nix/store/0csqp747mfw0v9n103abxgx611s6dkxm-icat.drv -building '/nix/store/0csqp747mfw0v9n103abxgx611s6dkxm-icat.drv'... + /nix/store/qg9f6zf0vwmvhz1w5i1fy2pw0l3wiqi9-icat.drv +building '/nix/store/qg9f6zf0vwmvhz1w5i1fy2pw0l3wiqi9-icat.drv'... unpacking sources -unpacking source archive /nix/store/y4750c9xljqy21b62a03z5xqvl3sd92q-source +unpacking source archive /nix/store/rx21f6fgnmxgp1sw0wbqll9wds4xc6v0-source source root is source patching sources configuring no configure script, doing nothing building build flags: SHELL=/nix/store/7q1b1bsmxi91zci6g8714rcljl620y7f-bash-5.2-p15/bin/bash -gcc -c -Wall -pedantic -std=c99 -D_DEFAULT_SOURCE `pkg-config --cflags imlib2` -o icat.o icat.c -/nix/store/7q1b1bsmxi91zci6g8714rcljl620y7f-bash-5.2-p15/bin/bash: line 1: pkg-config: command not found +gcc -c -Wall -pedantic -std=c99 -D_BSD_SOURCE -o icat.o icat.c +In file included from /nix/store/dpk5m64n0axk01fq8h2m0yl9hhpq2nqk-glibc-2.37-8-dev/include/bits/libc-header-start.h:33, + from /nix/store/dpk5m64n0axk01fq8h2m0yl9hhpq2nqk-glibc-2.37-8-dev/include/stdio.h:27, + from icat.c:31: +/nix/store/dpk5m64n0axk01fq8h2m0yl9hhpq2nqk-glibc-2.37-8-dev/include/features.h:195:3: warning: #warning "_BSD_SOURCE and _SVID_SOURCE are deprecated, use _DEFAULT_SOURCE" [8;;https://gcc.gnu.org/onlinedocs/gcc/Warning-Options.html#index-Wcpp-Wcpp8;;] + 195 | # warning "_BSD_SOURCE and _SVID_SOURCE are deprecated, use _DEFAULT_SOURCE" + | ^~~~~~~ In file included from icat.c:39: /nix/store/hkgbjcr182m3q9xs0j1qmp3dh08mbg31-imlib2-1.11.1-dev/include/Imlib2.h:45:10: fatal error: X11/Xlib.h: No such file or directory 45 | #include | ^~~~~~~~~~~~ compilation terminated. -make: *** [Makefile:20: icat.o] Error 1 -error: builder for '/nix/store/0csqp747mfw0v9n103abxgx611s6dkxm-icat.drv' failed with exit code 2; +make: *** [Makefile:16: icat.o] Error 1 +error: builder for '/nix/store/qg9f6zf0vwmvhz1w5i1fy2pw0l3wiqi9-icat.drv' failed with exit code 2; last 10 log lines: - > building - > build flags: SHELL=/nix/store/7q1b1bsmxi91zci6g8714rcljl620y7f-bash-5.2-p15/bin/bash - > gcc -c -Wall -pedantic -std=c99 -D_DEFAULT_SOURCE `pkg-config --cflags imlib2` -o icat.o icat.c - > /nix/store/7q1b1bsmxi91zci6g8714rcljl620y7f-bash-5.2-p15/bin/bash: line 1: pkg-config: command not found + > from icat.c:31: + > /nix/store/dpk5m64n0axk01fq8h2m0yl9hhpq2nqk-glibc-2.37-8-dev/include/features.h:195:3: warning: #warning "_BSD_SOURCE and _SVID_SOURCE are deprecated, use _DEFAULT_SOURCE" [8;;https://gcc.gnu.org/onlinedocs/gcc/Warning-Options.html#index-Wcpp-Wcpp8;;] + > 195 | # warning "_BSD_SOURCE and _SVID_SOURCE are deprecated, use _DEFAULT_SOURCE" + > | ^~~~~~~ > In file included from icat.c:39: > /nix/store/hkgbjcr182m3q9xs0j1qmp3dh08mbg31-imlib2-1.11.1-dev/include/Imlib2.h:45:10: fatal error: X11/Xlib.h: No such file or directory > 45 | #include > | ^~~~~~~~~~~~ > compilation terminated. - > make: *** [Makefile:20: icat.o] Error 1 - For full logs, run 'nix log /nix/store/0csqp747mfw0v9n103abxgx611s6dkxm-icat.drv'. + > make: *** [Makefile:16: icat.o] Error 1 + For full logs, run 'nix log /nix/store/qg9f6zf0vwmvhz1w5i1fy2pw0l3wiqi9-icat.drv'. ``` In Nixpkgs, `Xlib` lives in the `dev` output of `xorg.libX11`, which we can add to `buildInputs` again with `pkgs.xorg.libX11.dev`. To avoid repeating ourselves, we can add `pkgs` to the local scope in `buildInputs` by using the [`with` statement](https://nixos.org/guides/nix-pills/basics-of-language.html#idm140737320521984): @@ -434,53 +444,33 @@ Running our favorite command again, yet more errors arise: ```console $ nix-build -E 'with import {}; callPackage ./icat.nix {}' this derivation will be built: - /nix/store/0q6x7g7sz4pds3pgs8yb197fnf4r7rl2-icat.drv -building '/nix/store/0q6x7g7sz4pds3pgs8yb197fnf4r7rl2-icat.drv'... + /nix/store/p21p5zkbwg83dhmi0bn1yz5ka6phd47x-icat.drv +building '/nix/store/p21p5zkbwg83dhmi0bn1yz5ka6phd47x-icat.drv'... unpacking sources -unpacking source archive /nix/store/y4750c9xljqy21b62a03z5xqvl3sd92q-source +unpacking source archive /nix/store/rx21f6fgnmxgp1sw0wbqll9wds4xc6v0-source source root is source patching sources configuring no configure script, doing nothing building build flags: SHELL=/nix/store/7q1b1bsmxi91zci6g8714rcljl620y7f-bash-5.2-p15/bin/bash -gcc -c -Wall -pedantic -std=c99 -D_DEFAULT_SOURCE `pkg-config --cflags imlib2` -o icat.o icat.c -/nix/store/7q1b1bsmxi91zci6g8714rcljl620y7f-bash-5.2-p15/bin/bash: line 1: pkg-config: command not found +gcc -c -Wall -pedantic -std=c99 -D_BSD_SOURCE -o icat.o icat.c +In file included from /nix/store/dpk5m64n0axk01fq8h2m0yl9hhpq2nqk-glibc-2.37-8-dev/include/bits/libc-header-start.h:33, + from /nix/store/dpk5m64n0axk01fq8h2m0yl9hhpq2nqk-glibc-2.37-8-dev/include/stdio.h:27, + from icat.c:31: +/nix/store/dpk5m64n0axk01fq8h2m0yl9hhpq2nqk-glibc-2.37-8-dev/include/features.h:195:3: warning: #warning "_BSD_SOURCE and _SVID_SOURCE are deprecated, use _DEFAULT_SOURCE" [8;;https://gcc.gnu.org/onlinedocs/gcc/Warning-Options.html#index-Wcpp-Wcpp8;;] + 195 | # warning "_BSD_SOURCE and _SVID_SOURCE are deprecated, use _DEFAULT_SOURCE" + | ^~~~~~~ icat.c: In function 'main': icat.c:319:33: warning: ignoring return value of 'write' declared with attribute 'warn_unused_result' [8;;https://gcc.gnu.org/onlinedocs/gcc/Warning-Options.html#index-Wunused-result-Wunused-result8;;] 319 | write(tempfile, &buf, 1); | ^~~~~~~~~~~~~~~~~~~~~~~~ -gcc -o icat icat.o `pkg-config --libs imlib2` -/nix/store/7q1b1bsmxi91zci6g8714rcljl620y7f-bash-5.2-p15/bin/bash: line 1: pkg-config: command not found -/nix/store/dx8hynidprz3kf4ngcjipnwaxp6h229f-binutils-2.40/bin/ld: icat.o: in function `resize_image_if_necessary': -icat.c:(.text+0x12a): undefined reference to `imlib_create_cropped_scaled_image' -/nix/store/dx8hynidprz3kf4ngcjipnwaxp6h229f-binutils-2.40/bin/ld: icat.c:(.text+0x132): undefined reference to `imlib_free_image_and_decache' -/nix/store/dx8hynidprz3kf4ngcjipnwaxp6h229f-binutils-2.40/bin/ld: icat.c:(.text+0x13a): undefined reference to `imlib_context_set_image' -/nix/store/dx8hynidprz3kf4ngcjipnwaxp6h229f-binutils-2.40/bin/ld: icat.o: in function `main': -icat.c:(.text.startup+0x25b): undefined reference to `imlib_load_image_immediately' -/nix/store/dx8hynidprz3kf4ngcjipnwaxp6h229f-binutils-2.40/bin/ld: icat.c:(.text.startup+0x26c): undefined reference to `imlib_context_set_image' -/nix/store/dx8hynidprz3kf4ngcjipnwaxp6h229f-binutils-2.40/bin/ld: icat.c:(.text.startup+0x271): undefined reference to `imlib_image_get_width' -/nix/store/dx8hynidprz3kf4ngcjipnwaxp6h229f-binutils-2.40/bin/ld: icat.c:(.text.startup+0x27a): undefined reference to `imlib_image_get_height' -/nix/store/dx8hynidprz3kf4ngcjipnwaxp6h229f-binutils-2.40/bin/ld: icat.c:(.text.startup+0x2ea): undefined reference to `imlib_image_query_pixel' -/nix/store/dx8hynidprz3kf4ngcjipnwaxp6h229f-binutils-2.40/bin/ld: icat.c:(.text.startup+0x2f8): undefined reference to `imlib_image_query_pixel' -/nix/store/dx8hynidprz3kf4ngcjipnwaxp6h229f-binutils-2.40/bin/ld: icat.c:(.text.startup+0x33c): undefined reference to `imlib_free_image_and_decache' -collect2: error: ld returned 1 exit status -make: *** [Makefile:23: icat] Error 1 -error: builder for '/nix/store/0q6x7g7sz4pds3pgs8yb197fnf4r7rl2-icat.drv' failed with exit code 2; +gcc -o icat icat.o -lImlib2 +installing +install flags: SHELL=/nix/store/7q1b1bsmxi91zci6g8714rcljl620y7f-bash-5.2-p15/bin/bash install +make: *** No rule to make target 'install'. Stop. +error: builder for '/nix/store/p21p5zkbwg83dhmi0bn1yz5ka6phd47x-icat.drv' failed with exit code 2; last 10 log lines: - > /nix/store/dx8hynidprz3kf4ngcjipnwaxp6h229f-binutils-2.40/bin/ld: icat.o: in function `main': - > icat.c:(.text.startup+0x25b): undefined reference to `imlib_load_image_immediately' - > /nix/store/dx8hynidprz3kf4ngcjipnwaxp6h229f-binutils-2.40/bin/ld: icat.c:(.text.startup+0x26c): undefined reference to `imlib_context_set_image' - > /nix/store/dx8hynidprz3kf4ngcjipnwaxp6h229f-binutils-2.40/bin/ld: icat.c:(.text.startup+0x271): undefined reference to `imlib_image_get_width' - > /nix/store/dx8hynidprz3kf4ngcjipnwaxp6h229f-binutils-2.40/bin/ld: icat.c:(.text.startup+0x27a): undefined reference to `imlib_image_get_height' - > /nix/store/dx8hynidprz3kf4ngcjipnwaxp6h229f-binutils-2.40/bin/ld: icat.c:(.text.startup+0x2ea): undefined reference to `imlib_image_query_pixel' - > /nix/store/dx8hynidprz3kf4ngcjipnwaxp6h229f-binutils-2.40/bin/ld: icat.c:(.text.startup+0x2f8): undefined reference to `imlib_image_query_pixel' - > /nix/store/dx8hynidprz3kf4ngcjipnwaxp6h229f-binutils-2.40/bin/ld: icat.c:(.text.startup+0x33c): undefined reference to `imlib_free_image_and_decache' - > collect2: error: ld returned 1 exit status - > make: *** [Makefile:23: icat] Error 1 - For full logs, run 'nix log /nix/store/0q6x7g7sz4pds3pgs8yb197fnf4r7rl2-icat.drv'. -``` - There are several issues here, but the first one we can solve is `/nix/store/7q1b1bsmxi91zci6g8714rcljl620y7f-bash-5.2-p15/bin/bash: line 1: pkg-config: command not found`. According to the [Nixpkgs Manual](https://nixos.org/manual/nixpkgs/stable/#ssec-stdenv-dependencies), we should add dependencies to `buildInputs` if they're going to be copied or linked into the final output, or otherwise used somehow at runtime, but we should add dependencies to the `nativeBuildInputs` list if those dependencies are used at *build* time. `pkg-config` isn't needed after we build `icat`, so we'll add it to `nativeBuildInputs`: ```nix @@ -526,14 +516,17 @@ $ make In the current `master` branch of `icat`, a warning is thrown when building: -```console -[nix-shell:~/icat]$ make -gcc -c -Wall -pedantic -std=c99 -D_DEFAULT_SOURCE `pkg-config --cflags imlib2` -o icat.o icat.c -icat.c: In function ‘main’: -icat.c:319:33: warning: ignoring return value of ‘write’ declared with attribute ‘warn_unused_result’ [-Wunused-result] - 319 | write(tempfile, &buf, 1); - | ^~~~~~~~~~~~~~~~~~~~~~~~ -gcc -o icat icat.o `pkg-config --libs imlib2` + > 195 | # warning "_BSD_SOURCE and _SVID_SOURCE are deprecated, use _DEFAULT_SOURCE" + > | ^~~~~~~ + > icat.c: In function 'main': + > icat.c:319:33: warning: ignoring return value of 'write' declared with attribute 'warn_unused_result' [8;;https://gcc.gnu.org/onlinedocs/gcc/Warning-Options.html#index-Wunused-result-Wunused-result8;;] + > 319 | write(tempfile, &buf, 1); + > | ^~~~~~~~~~~~~~~~~~~~~~~~ + > gcc -o icat icat.o -lImlib2 + > installing + > install flags: SHELL=/nix/store/7q1b1bsmxi91zci6g8714rcljl620y7f-bash-5.2-p15/bin/bash install + > make: *** No rule to make target 'install'. Stop. + For full logs, run 'nix log /nix/store/p21p5zkbwg83dhmi0bn1yz5ka6phd47x-icat.drv'. ``` However, this does not prevent the binary from being produced; an `icat` executable is now present in the local directory, and it's up to us to decide what to do with it. From 319593ae83b07fe436c043c3364b114ad49186a1 Mon Sep 17 00:00:00 2001 From: alex Date: Thu, 20 Jul 2023 14:06:11 -0500 Subject: [PATCH 15/74] shorten some derivation examples and fix whitespace --- .../tutorials/packaging-existing-software.md | 30 +++++++++++-------- 1 file changed, 17 insertions(+), 13 deletions(-) diff --git a/source/tutorials/packaging-existing-software.md b/source/tutorials/packaging-existing-software.md index 229333cbc..a8eccfc90 100644 --- a/source/tutorials/packaging-existing-software.md +++ b/source/tutorials/packaging-existing-software.md @@ -123,6 +123,7 @@ Let's update the file again to add a `name`: stdenv.mkDerivation { name = "hello"; + src = builtins.fetchTarball { url = "https://ftp.gnu.org/gnu/hello/hello-2.12.1.tar.gz"; sha256 = lib.fakeSha256; @@ -167,6 +168,7 @@ As expected, Nix complained at us for lying about the file hash, and helpfully p stdenv.mkDerivation { name = "hello"; + src = builtins.fetchTarball { url = "https://ftp.gnu.org/gnu/hello/hello-2.12.1.tar.gz"; sha256 = "0xw6cr5jgi1ir13q6apvrivwmmpr5j8vbymp0x6ll0kcv6366hnn"; @@ -232,11 +234,9 @@ We'll start by copying `hello.nix` from the previous section to a new file, `ica stdenv.mkDerivation { name = "icat"; - src = builtins.fetchFromGitHub { - owner = "atextor"; - repo = "icat"; - rev = "master"; - sha256 = lib.fakeSha256; + + src = builtins.fetchTarball { + ... }; } ``` @@ -252,11 +252,9 @@ Now we'll download the source code. `icat`'s upstream repository is hosted on [G stdenv.mkDerivation { name = "icat"; + src = pkgs.fetchFromGitHub { - owner = "atextor"; - repo = "icat"; - rev = "master"; - sha256 = lib.fakeSha256; + ... }; } ``` @@ -289,6 +287,7 @@ Now we can supply the correct hash to `fetchFromGitHub`: stdenv.mkDerivation { name = "icat"; + src = pkgs.fetchFromGitHub { owner = "atextor"; repo = "icat"; @@ -359,6 +358,7 @@ Because `callPackage` is used to provide all necessary inputs in `nixpkgs` as we stdenv.mkDerivation { name = "icat"; + src = pkgs.fetchFromGitHub { owner = "atextor"; repo = "icat"; @@ -368,7 +368,6 @@ stdenv.mkDerivation { buildInputs = [ imlib2 ]; } - ``` Another error, but we get further this time: @@ -427,6 +426,7 @@ In Nixpkgs, `Xlib` lives in the `dev` output of `xorg.libX11`, which we can add stdenv.mkDerivation { name = "icat"; + src = pkgs.fetchFromGitHub { owner = "atextor"; repo = "icat"; @@ -434,13 +434,17 @@ stdenv.mkDerivation { sha256 = "0wyy2ksxp95vnh71ybj1bbmqd5ggp13x3mk37pzr99ljs9awy8ka"; }; - buildInputs = with pkgs; [ imlib2 xorg.libX11.dev ]; + buildInputs = [ imlib2 xorg.libX11.dev ]; } ``` -### `buildInputs` and `nativeBuildInputs` -Running our favorite command again, yet more errors arise: +:::{note} +We only added the top-level `xorg` derivation to the input attrset, rather than the full `xorg.libX11.dev` which would cause a syntax error. Because Nix is lazily-evaluated, including the dependency this way doesn't actually include all of `xorg` into our build context. +::: + +### `buildInputs` and `nativeBuildInputs` +Running our favorite command again: ```console $ nix-build -E 'with import {}; callPackage ./icat.nix {}' this derivation will be built: From 5ee2b986c9e785cd25e30b12389428187e211922 Mon Sep 17 00:00:00 2001 From: alex Date: Thu, 20 Jul 2023 14:06:55 -0500 Subject: [PATCH 16/74] remove description of nixpkgs import/callPackage expression --- source/tutorials/packaging-existing-software.md | 3 --- 1 file changed, 3 deletions(-) diff --git a/source/tutorials/packaging-existing-software.md b/source/tutorials/packaging-existing-software.md index a8eccfc90..a32b1bdeb 100644 --- a/source/tutorials/packaging-existing-software.md +++ b/source/tutorials/packaging-existing-software.md @@ -94,13 +94,10 @@ We'll use the following expression: ```console with import {}; callPackage ./hello.nix {} -^ ^ ^ ^ ^ ^ ^ -1 2 3 4 5 6 7 ``` `callPackage` automatically passes attributes from `pkgs` to the given function, if they match attributes required by that function's argument attrset. Here, `callPackage` will supply `pkgs`, `lib`, and `stdenv`. -From `nixpkgs`, we use the `callPackage` function (5) to import our `hello.nix` (6). Since the `hello.nix` derivation [is a function](https://nixos.org/manual/nix/stable/language/derivations.html) which takes two arguments [it will already be passed](), we also apply it to an empty attribute set (7). Although `hello.nix` takes a `pkgs` argument, we don't need to apply the function to `nixpkgs`, because the `with` statement has already included the `nixpkgs` we just imported into the local context. Let's run this now: From f17f33ec690b4189b91ffe3d3a3ce11322862d05 Mon Sep 17 00:00:00 2001 From: alex Date: Thu, 20 Jul 2023 14:08:58 -0500 Subject: [PATCH 17/74] pkg-config no longer needed --- .../tutorials/packaging-existing-software.md | 23 ------------------- 1 file changed, 23 deletions(-) diff --git a/source/tutorials/packaging-existing-software.md b/source/tutorials/packaging-existing-software.md index a32b1bdeb..95a071738 100644 --- a/source/tutorials/packaging-existing-software.md +++ b/source/tutorials/packaging-existing-software.md @@ -472,28 +472,6 @@ install flags: SHELL=/nix/store/7q1b1bsmxi91zci6g8714rcljl620y7f-bash-5.2-p15/bi make: *** No rule to make target 'install'. Stop. error: builder for '/nix/store/p21p5zkbwg83dhmi0bn1yz5ka6phd47x-icat.drv' failed with exit code 2; last 10 log lines: -There are several issues here, but the first one we can solve is `/nix/store/7q1b1bsmxi91zci6g8714rcljl620y7f-bash-5.2-p15/bin/bash: line 1: pkg-config: command not found`. According to the [Nixpkgs Manual](https://nixos.org/manual/nixpkgs/stable/#ssec-stdenv-dependencies), we should add dependencies to `buildInputs` if they're going to be copied or linked into the final output, or otherwise used somehow at runtime, but we should add dependencies to the `nativeBuildInputs` list if those dependencies are used at *build* time. `pkg-config` isn't needed after we build `icat`, so we'll add it to `nativeBuildInputs`: - -```nix -{ pkgs -, lib -, stdenv -}: - -stdenv.mkDerivation { - name = "icat"; - src = pkgs.fetchFromGitHub { - owner = "atextor"; - repo = "icat"; - rev = "master"; - sha256 = "sha256-b/2mRzCTyGkz2I1U+leUhspvW77VcHN7Awp+BVdVNRM="; - }; - - nativeBuildInputs = with pkgs; [ pkg-config ]; - buildInputs = with pkgs; [ imlib2 xorg.libX11.dev ]; -} -``` - ### Debugging with a Development Shell This solves some of the errors we just saw, but not all; the `ld` error produced by all the undefined references is gone, but we still see a non-zero `make` return value: `make: *** No rule to make target 'install'`. @@ -599,7 +577,6 @@ stdenv.mkDerivation { sha256 = "0wyy2ksxp95vnh71ybj1bbmqd5ggp13x3mk37pzr99ljs9awy8ka"; }; - nativeBuildInputs = with pkgs; [ pkg-config ]; buildInputs = with pkgs; [ imlib2 xorg.libX11.dev ]; installPhase = '' From 12e41e3a405b315106bb9d93ce2692e2d24d2ceb Mon Sep 17 00:00:00 2001 From: alex Date: Thu, 20 Jul 2023 14:13:09 -0500 Subject: [PATCH 18/74] remove contributing, metadata, and devshell sections --- .../tutorials/packaging-existing-software.md | 87 ------------------- 1 file changed, 87 deletions(-) diff --git a/source/tutorials/packaging-existing-software.md b/source/tutorials/packaging-existing-software.md index 95a071738..e7c537b7c 100644 --- a/source/tutorials/packaging-existing-software.md +++ b/source/tutorials/packaging-existing-software.md @@ -472,29 +472,6 @@ install flags: SHELL=/nix/store/7q1b1bsmxi91zci6g8714rcljl620y7f-bash-5.2-p15/bi make: *** No rule to make target 'install'. Stop. error: builder for '/nix/store/p21p5zkbwg83dhmi0bn1yz5ka6phd47x-icat.drv' failed with exit code 2; last 10 log lines: -### Debugging with a Development Shell -This solves some of the errors we just saw, but not all; the `ld` error produced by all the undefined references is gone, but we still see a non-zero `make` return value: `make: *** No rule to make target 'install'`. - -Nix is automatically working with the `Makefile` that comes with `icat`, which indeed lacks an `install` target. The `README` in the `icat` repository only mentions using `make` to build the tool, presumably leaving installation up to us. We've now discovered one limit to what Nix can do for us automatically: it doesn't read minds. Fortunately, it does still make the fix quite straightforward to implement. - -If you haven't read the tutorials on creating [ad-hoc](https://nix.dev/tutorials/first-steps/dev-environment) or [declarative](https://nix.dev/tutorials/first-steps/declarative-and-reproducible-developer-environments) development environments, do that now before proceeding through the rest of this tutorial; dropping into a `nix-shell` is a crucial component in the Nix user toolbox, and indispensible for debugging. - -To enter a useful development shell, we'll pass the dependencies from `nativeBuildInputs` and `buildInputs` to `nix-shell -p`. We'll also make sure to include `git`, so we can clone the `icat` GitHub repository: - -```console -$ nix-shell -p pkg-config imlib2 xorg.libX11.dev git -``` - -After many lines of output about Nix copying dependencies, we can use the following commands to retrieve and build the `icat` source code: - -```console -$ git clone https://github.com/atextor/icat -$ cd icat -$ make -``` - -In the current `master` branch of `icat`, a warning is thrown when building: - > 195 | # warning "_BSD_SOURCE and _SVID_SOURCE are deprecated, use _DEFAULT_SOURCE" > | ^~~~~~~ > icat.c: In function 'main': @@ -508,7 +485,6 @@ In the current `master` branch of `icat`, a warning is thrown when building: For full logs, run 'nix log /nix/store/p21p5zkbwg83dhmi0bn1yz5ka6phd47x-icat.drv'. ``` -However, this does not prevent the binary from being produced; an `icat` executable is now present in the local directory, and it's up to us to decide what to do with it. ### installPhase In order to make packages available for other packages to depend on, Nix copies everything to the Nix store (at `/nix/store`), and symlinks them from there into build contexts and development environments. @@ -528,48 +504,7 @@ The `icat` executable is only used at runtime, and isn't a compile-time input fo stdenv.mkDerivation { name = "icat"; - src = pkgs.fetchFromGitHub { - owner = "atextor"; - repo = "icat"; - rev = "master"; - sha256 = "sha256-b/2mRzCTyGkz2I1U+leUhspvW77VcHN7Awp+BVdVNRM="; - }; - - nativeBuildInputs = with pkgs; [ pkg-config ]; - buildInputs = with pkgs; [ imlib2 xorg.libX11.dev ]; - - installPhase = '' - mkdir -p $out/bin - cp icat $out/bin - ''; -} -``` - -After running our `nix-build` command one last time, we can `ls` in the local directory to find a `result` symlink to the Nix store, with `result/bin/icat` the executable we built. Success! - -## Contributing our Work -Now that we've packaged `icat`, it's time to prepare it for submission upstream to Nixpkgs. - -### Building a Release Version -Our `icat.nix` definition uses the `master` revision of the upstream repository, which is suitable for individual use but not a great practice for submission to Nixpkgs; it would be better to use a fixed revision, corresponding to a particular release version of the software, at least so maintainers (perhaps you!) could easily check when this package should be updated. -The upstream GitHub repository has [several tags available](https://github.com/atextor/icat/tags), which correspond to released versions. We'll modify our existing `icat.nix` to download and build the latest tag instead of what's available on `master`. This time, instead of using the `lib.fakeSha256` trick, we'll use [`nix-prefetch-url`](https://nixos.org/manual/nix/stable/command-ref/nix-prefetch-url.html) to retrieve the hash we need: - -```console -$ nix-prefetch-url --unpack https://github.com/atextor/icat/archive/refs/tags/v0.5.tar.gz -path is '/nix/store/p8jl1jlqxcsc7ryiazbpm7c1mqb6848b-v0.5.tar.gz' -0wyy2ksxp95vnh71ybj1bbmqd5ggp13x3mk37pzr99ljs9awy8ka -``` - -```nix -# icat.nix -{ pkgs -, lib -, stdenv -}: - -stdenv.mkDerivation { - name = "icat"; src = pkgs.fetchFromGitHub { owner = "atextor"; repo = "icat"; @@ -654,28 +589,6 @@ Nix automatically determined the `buildPhase` information for our `icat` package ... ``` -### Package Metadata -By convention, all packages in Nixpkgs have a `meta` attribute in their derivation, which contains information like a description of the package, the homepage of the project it belongs to, the software license, the platforms the package can be built for, and a list of Nixpkgs maintainers for the package. In this case, I'm the contributing user, so I'll add myself to the maintainers list for this package. - -:::{note} -Before contributing your first package, you must add your information to `nixpkgs/maintainers/maintainers-list.nix`, following the instructions [here](https://nixos.org/manual/nixpkgs/stable/#var-meta-maintainers). -::: - -Before we contribute our package, we should add this metadata to the `meta` attribute passed to `mkDerivation`, following the [contribution guidelines](https://nixos.org/manual/nixpkgs/stable/#reviewing-contributions-new-packages): - -```nix -# icat.nix -... - meta = with lib; { - description = "icat (Image cat) outputs images in 256-color capable terminals."; - homepage = "https://github.com/atextor/icat"; - license = licenses.bsdOriginal; - platforms = platforms.unix; - maintainers = [ maintainers.proofconstruction ]; - }; -... -``` - ```console ``` From f1dcba406d2175d3a3f9c7ff3f3fbfee525ab1ff Mon Sep 17 00:00:00 2001 From: alex Date: Thu, 20 Jul 2023 14:13:59 -0500 Subject: [PATCH 19/74] remove section building package without phase hooks --- .../tutorials/packaging-existing-software.md | 47 ------------------- 1 file changed, 47 deletions(-) diff --git a/source/tutorials/packaging-existing-software.md b/source/tutorials/packaging-existing-software.md index e7c537b7c..7e9098f5e 100644 --- a/source/tutorials/packaging-existing-software.md +++ b/source/tutorials/packaging-existing-software.md @@ -521,53 +521,6 @@ stdenv.mkDerivation { } ``` -And it builds! -```console -$ nix-build -E 'with import {}; callPackage ./icat.nix {}' -these 2 derivations will be built: - /nix/store/vvjyrngklzxbcsfiyp4hr1z2qcdqm8j7-source.drv - /nix/store/x6h1kfd4h16vhj0cxlakrm5igbbbz7v3-icat.drv -building '/nix/store/vvjyrngklzxbcsfiyp4hr1z2qcdqm8j7-source.drv'... - -trying https://github.com/atextor/icat/archive/v0.5.tar.gz - % Total % Received % Xferd Average Speed Time Time Time Current - Dload Upload Total Spent Left Speed - 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 -100 46232 0 46232 0 0 60947 0 --:--:-- --:--:-- --:--:-- 60947 -unpacking source archive /build/v0.5.tar.gz -building '/nix/store/x6h1kfd4h16vhj0cxlakrm5igbbbz7v3-icat.drv'... -unpacking sources -unpacking source archive /nix/store/rx21f6fgnmxgp1sw0wbqll9wds4xc6v0-source -source root is source -patching sources -configuring -no configure script, doing nothing -building -build flags: SHELL=/nix/store/7q1b1bsmxi91zci6g8714rcljl620y7f-bash-5.2-p15/bin/bash -gcc -c -Wall -pedantic -std=c99 -D_BSD_SOURCE -o icat.o icat.c -In file included from /nix/store/dpk5m64n0axk01fq8h2m0yl9hhpq2nqk-glibc-2.37-8-dev/include/bits/libc-header-start.h:33, - from /nix/store/dpk5m64n0axk01fq8h2m0yl9hhpq2nqk-glibc-2.37-8-dev/include/stdio.h:27, - from icat.c:31: -/nix/store/dpk5m64n0axk01fq8h2m0yl9hhpq2nqk-glibc-2.37-8-dev/include/features.h:195:3: warning: #warning "_BSD_SOURCE and _SVID_SOURCE are deprecated, use _DEFAULT_SOURCE" [8;;https://gcc.gnu.org/onlinedocs/gcc/Warning-Options.html#index-Wcpp-Wcpp8;;] - 195 | # warning "_BSD_SOURCE and _SVID_SOURCE are deprecated, use _DEFAULT_SOURCE" - | ^~~~~~~ -icat.c: In function 'main': -icat.c:319:33: warning: ignoring return value of 'write' declared with attribute 'warn_unused_result' [8;;https://gcc.gnu.org/onlinedocs/gcc/Warning-Options.html#index-Wunused-result-Wunused-result8;;] - 319 | write(tempfile, &buf, 1); - | ^~~~~~~~~~~~~~~~~~~~~~~~ -gcc -o icat icat.o -lImlib2 -installing -post-installation fixup -shrinking RPATHs of ELF executables and libraries in /nix/store/g6w508vxwr3df25dnl4k3xvcr4pqxprj-icat -shrinking /nix/store/g6w508vxwr3df25dnl4k3xvcr4pqxprj-icat/bin/icat -checking for references to /build/ in /nix/store/g6w508vxwr3df25dnl4k3xvcr4pqxprj-icat... -patching script interpreter paths in /nix/store/g6w508vxwr3df25dnl4k3xvcr4pqxprj-icat -stripping (with command strip and flags -S -p) in /nix/store/g6w508vxwr3df25dnl4k3xvcr4pqxprj-icat/bin -/nix/store/g6w508vxwr3df25dnl4k3xvcr4pqxprj-icat -``` - -We still see the unused-result warning thrown by the compiler, but the package successfully built, and the very last line of output tells us where Nix put the result. - ### Phases and Hooks Nix package derivations are separated into [phases](https://nixos.org/manual/nixpkgs/unstable/#sec-stdenv-phases), each of which is intended to control some aspect of the build process. From 4f6d0fa176a8937dd2206c511927718b4e34d1c0 Mon Sep 17 00:00:00 2001 From: alex Date: Thu, 20 Jul 2023 14:16:04 -0500 Subject: [PATCH 20/74] improve end of icat packaging, add ls output --- source/tutorials/packaging-existing-software.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/source/tutorials/packaging-existing-software.md b/source/tutorials/packaging-existing-software.md index 7e9098f5e..19d84487e 100644 --- a/source/tutorials/packaging-existing-software.md +++ b/source/tutorials/packaging-existing-software.md @@ -542,7 +542,11 @@ Nix automatically determined the `buildPhase` information for our `icat` package ... ``` +Running our `nix-build` command once more will finally do what we want, and more safely than before; we can `ls` in the local directory to find a `result` symlink to a location in the Nix store: ```console +$ ls +hello.nix icat.nix result ``` +`result/bin/icat` is the executable we built previously. Success! From 44d8c15f3e53331c03ef488039cceff35f4da68b Mon Sep 17 00:00:00 2001 From: alex Date: Thu, 20 Jul 2023 14:16:48 -0500 Subject: [PATCH 21/74] expand section on adding Xlib dependency --- source/tutorials/packaging-existing-software.md | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/source/tutorials/packaging-existing-software.md b/source/tutorials/packaging-existing-software.md index 19d84487e..e4f1c5278 100644 --- a/source/tutorials/packaging-existing-software.md +++ b/source/tutorials/packaging-existing-software.md @@ -410,7 +410,15 @@ error: builder for '/nix/store/qg9f6zf0vwmvhz1w5i1fy2pw0l3wiqi9-icat.drv' failed For full logs, run 'nix log /nix/store/qg9f6zf0vwmvhz1w5i1fy2pw0l3wiqi9-icat.drv'. ``` -In Nixpkgs, `Xlib` lives in the `dev` output of `xorg.libX11`, which we can add to `buildInputs` again with `pkgs.xorg.libX11.dev`. To avoid repeating ourselves, we can add `pkgs` to the local scope in `buildInputs` by using the [`with` statement](https://nixos.org/guides/nix-pills/basics-of-language.html#idm140737320521984): +We can see a few warnings which should be corrected in the upstream code, but the important bit for our purposes is `fatal error: X11/Xlib.h: No such file or directory`: we're missing another dependency. + +In addition to the widespread practice of prefixing a project name with `lib` to indicate the libraries of that project, in Nixpkgs it's also common to separate headers, libraries, binaries, and documentation into different output attributes of a given [derivation](https://nixos.org/manual/nix/stable/language/derivations.html). + +:::{note} +Determining from where to source a dependency is currently a somewhat-involved process: it helps to become familiar with searching the `nixpkgs` source for keywords, in addition to checking discussion platforms like [the official NixOS Discourse](https://discourse.nixos.org). +::: + +We need the `Xlib.h` headers from the `X11` C package, the Nixpkgs derivation for which is `libX11`, available in the `xorg` package set. The `Xlib` headers in turn live in the `dev` output of `xorg.libX11`. We'll add this to our derivation's input attribute set and to `buildInputs`: ```nix # icat.nix From a0091bac2451eaecb441e4c554f50cc619f7bb03 Mon Sep 17 00:00:00 2001 From: alex Date: Thu, 20 Jul 2023 14:17:27 -0500 Subject: [PATCH 22/74] improve discussion of phase hooks --- source/tutorials/packaging-existing-software.md | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/source/tutorials/packaging-existing-software.md b/source/tutorials/packaging-existing-software.md index e4f1c5278..23b4a6d0d 100644 --- a/source/tutorials/packaging-existing-software.md +++ b/source/tutorials/packaging-existing-software.md @@ -532,11 +532,15 @@ stdenv.mkDerivation { ### Phases and Hooks Nix package derivations are separated into [phases](https://nixos.org/manual/nixpkgs/unstable/#sec-stdenv-phases), each of which is intended to control some aspect of the build process. -During derivation realisation, there are a number of shell functions ("hooks", in `nixpkgs`) which may execute in each derivation phase, which do things like set variables, source files, create directories, and so on. These are run both before and after each phase, controlling the build environment and helping to prevent environment-modifying behavior defined within packages from creating sources of nondeterminism within and between Nix derivations. +We saw earlier how the `stdenv` expected the project's `Makefile` to have an `install` target, and failed when it didn't. To fix this, we defined a custom `installPhase`, containing instructions for copying the `icat` binary to the correct output location, in effect installing it. -It's good practice when packaging for `nixpkgs` to include calls to these hooks in the derivation phases you define, even when you don't make direct use of them; this facilitates easy [overriding](https://nixos.org/manual/nixpkgs/stable/#chap-overrides) of specific parts of the derivation later, in addition to the previously-mentioned reproducibility benefits. +Up to that point, the `stdenv` automatically determined the `buildPhase` information for our `icat` package. -Nix automatically determined the `buildPhase` information for our `icat` package, but we needed to define a custom `installPhase` which we should now adjust to call the appropriate hooks: +During derivation realisation, there are a number of shell functions ("hooks", in `nixpkgs`) which may execute in each derivation phase, which do things like set variables, source files, create directories, and so on. These are specific to each phase, and run both before and after that phase's execution, controlling the build environment and helping to prevent environment-modifying behavior defined within packages from creating sources of nondeterminism within and between Nix derivations. + +It's good practice when packaging software with Nix to include calls to these hooks in the derivation phases you define, even when you don't make direct use of them; this facilitates easy [overriding](https://nixos.org/manual/nixpkgs/stable/#chap-overrides) of specific parts of the derivation later, in addition to the previously-mentioned reproducibility benefits. + +We should now adjust our `installPhase` to call the appropriate hooks: ```nix # icat.nix From d8066b64e7e4ede9e9f916f1deeac48f7fd9da97 Mon Sep 17 00:00:00 2001 From: alex Date: Thu, 20 Jul 2023 14:19:29 -0500 Subject: [PATCH 23/74] better motivate custom installPhase --- source/tutorials/packaging-existing-software.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/source/tutorials/packaging-existing-software.md b/source/tutorials/packaging-existing-software.md index 23b4a6d0d..056b6aae1 100644 --- a/source/tutorials/packaging-existing-software.md +++ b/source/tutorials/packaging-existing-software.md @@ -493,13 +493,14 @@ error: builder for '/nix/store/p21p5zkbwg83dhmi0bn1yz5ka6phd47x-icat.drv' failed For full logs, run 'nix log /nix/store/p21p5zkbwg83dhmi0bn1yz5ka6phd47x-icat.drv'. ``` +We've solved the missing dependency, but have another problem: `make: *** No rule to make target 'install'. Stop.` ### installPhase -In order to make packages available for other packages to depend on, Nix copies everything to the Nix store (at `/nix/store`), and symlinks them from there into build contexts and development environments. +The `stdenv` is automatically working with the `Makefile` that comes with `icat`: we can see in the console output that `configure` and `make` are executed without issue, so the `icat` binary is compiling successfully. The failure occurs when the `stdenv` attempts to run `make install`: the `Makefile` included in the project happens to lack an `install` target, and the `README` in the `icat` repository only mentions using `make` to build the tool, leaving the installation step up to us. -The `Makefile` doesn't provide an installation step, so we must produce one for our derivation, using the [`installPhase` attribute](https://nixos.org/manual/nixpkgs/stable/#ssec-install-phase), which contains a list of command strings to execute to accomplish the installation. +To add this step to our derivation, we use the [`installPhase` attribute](https://nixos.org/manual/nixpkgs/stable/#ssec-install-phase), which contains a list of command strings to execute to perform the installation. -The `icat` executable is only used at runtime, and isn't a compile-time input for anything else at this point, so we only need to concern ourselves with the `bin` output. In Nix, the result of a build is copied to a location stored in the `$out` variable accessible in the derivation's component scripts; we'll create a `bin` directory within that, and then copy our `icat` executable there: +Because the `make` step completes successfully, the `icat` executable is available in the build directory, and we only need to copy it from there to the output directory. In Nix, this location is stored in the `$out` variable, accessible in the derivation's component scripts; we'll create a `bin` directory within that and copy our `icat` binary there: ```nix # icat.nix From 20ae449be734577839e7ec18b83632f1bd94590a Mon Sep 17 00:00:00 2001 From: alex Date: Thu, 20 Jul 2023 14:20:42 -0500 Subject: [PATCH 24/74] improve clarity and precision --- source/tutorials/packaging-existing-software.md | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) diff --git a/source/tutorials/packaging-existing-software.md b/source/tutorials/packaging-existing-software.md index 056b6aae1..988e31a8a 100644 --- a/source/tutorials/packaging-existing-software.md +++ b/source/tutorials/packaging-existing-software.md @@ -90,7 +90,7 @@ error: cannot evaluate a function that has an argument without a value ('pkgs') ### Your New Favorite Command In order to pass the `pkgs` argument to our derivation, we'll need to import `nixpkgs` in another Nix expression. The `nix-build` command lets us pass whole expressions as an argument following the `-E/--expr` flag. -We'll use the following expression: +We'll pass the following expression to `nix-build`: ```console with import {}; callPackage ./hello.nix {} @@ -105,7 +105,7 @@ Let's run this now: $ nix-build -E 'with import {}; callPackage ./hello.nix {}' error: derivation name missing ``` -Progress! The new failure occurs with the *derivation*, further down in the file than the initial error on line 2 about the `pkgs` argument not having a value; we successfully resolved the previous error by importing `nixpkgs` in the expression we passed to `nix-build`. +Progress! The new failure occurs with the *derivation*, further down in the file than the initial error on line 2 about the `pkgs` argument not having a value; we successfully resolved the previous error by changing the expression passed to `nix-build`. ### Naming a Derivation Every derivation needs a `name` attribute, which must either be set directly or constructed by `mkDerivation` from `pname` and `version` attributes, if they exist. @@ -213,7 +213,9 @@ $ ./result/bin/hello Hello, world! ``` -We've successfully packaged our first program with Nix! The experience was a little bit *too* magical though, so up next we'll package another piece of software which has external dependencies and a different means of building, which will require us to lean more on `mkDerivation`. +We've successfully packaged our first program with Nix! + +Up next, we'll package another piece of software which has external dependencies that present new challenges, requiring us to lean more on `mkDerivation`. ## Something Bigger The `hello` program is a simple and common place to start packaging, but it's not very useful or interesting, so we can't stop there. @@ -295,7 +297,7 @@ stdenv.mkDerivation { ``` ### Missing Dependencies -Now we run into an entirely new issue: +Running our previous `nix-build` invocation, we run into an entirely new issue: ```console $ nix-build -E 'with import {}; callPackage ./icat.nix {}' @@ -337,7 +339,7 @@ error: builder for '/nix/store/al2wld63c66p3ln0rxqlkqqrqpspnicj-icat.drv' failed For full logs, run 'nix log /nix/store/al2wld63c66p3ln0rxqlkqqrqpspnicj-icat.drv'. ``` -Finally, a compiler error! We've successfully pulled the `icat` source from GitHub, and Nix tried to build what it found, but is missing a dependency: the `imlib2` header. If we [search for `imlib2` on search.nixos.org](https://search.nixos.org/packages?channel=23.05&from=0&size=50&sort=relevance&type=packages&query=imlib2), we'll find that `imlib2` is already in `nixpkgs`. +Finally, a compiler error! We've successfully pulled the `icat` source from GitHub, and Nix tried to build what it found, but compilation failed due to a missing dependency: the `imlib2` header. If we [search for `imlib2` on search.nixos.org](https://search.nixos.org/packages?channel=23.05&from=0&size=50&sort=relevance&type=packages&query=imlib2), we'll find that `imlib2` is already in `nixpkgs`. We can add this package to our build environment by either - adding `imlib2` to the set of inputs to the expression in `icat.nix`, and then adding `imlib2` to the list of `buildInputs` in `stdenv.mkDerivation`, or From 03d6e30d12c08f8a9f534e582b96185d9d680f5e Mon Sep 17 00:00:00 2001 From: Alexander Groleau Date: Fri, 4 Aug 2023 19:11:55 -0500 Subject: [PATCH 25/74] "package" as derivation result --- source/tutorials/packaging-existing-software.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/source/tutorials/packaging-existing-software.md b/source/tutorials/packaging-existing-software.md index 988e31a8a..12f054a09 100644 --- a/source/tutorials/packaging-existing-software.md +++ b/source/tutorials/packaging-existing-software.md @@ -28,7 +28,7 @@ Along the way, we'll encounter and address Nix error messages, build failures, a ## Packages in Nix Before we proceed, an important point of clarification: we conventionally use the term "package" by analogy to other systems, although this term is not a proper concept in Nix. -For the purposes of this tutorial, by "package" we mean "a Nix function which takes an attribute set of 'dependencies' and produces a derivation", where "dependencies" could be other packages or configuration parameters. +For the purposes of this tutorial, by "package" we mean something like "result of a derivation"; this is the artifact you or others will use, as a consequence of having "packaged existing software with Nix". ## A Simple Project To start, we'll write a skeleton derivation, updating this as we go: From e4930ffe1a318f0f038ef40ad2682e748e95f564 Mon Sep 17 00:00:00 2001 From: Alexander Groleau Date: Fri, 4 Aug 2023 19:17:51 -0500 Subject: [PATCH 26/74] mkDerivation tutorial doesn't exist yet --- source/tutorials/packaging-existing-software.md | 1 - 1 file changed, 1 deletion(-) diff --git a/source/tutorials/packaging-existing-software.md b/source/tutorials/packaging-existing-software.md index 12f054a09..1bc41aa7f 100644 --- a/source/tutorials/packaging-existing-software.md +++ b/source/tutorials/packaging-existing-software.md @@ -15,7 +15,6 @@ One of Nix's primary use-cases is in addressing common difficulties encountered In the long term, Nix helps tremendously in alleviating that stress, but when *first* (re)packaging existing software with Nix, it's common to encounter missing dependencies preventing builds from succeeding. -If you haven't already read the tutorial on making a derivation, please go do so before reading this! In this tutorial, we'll see how to create Nix derivations to package C/C++ software, taking advantage of the [`nixpkgs` `stdenv`](https://nixos.org/manual/nixpkgs/stable/#chap-stdenv) which automates much of the work of building self-contained C/C++ packages. From 8762a1c6f91bdd95f87b619d244326903bedf855 Mon Sep 17 00:00:00 2001 From: Alexander Groleau Date: Mon, 7 Aug 2023 10:11:19 -0500 Subject: [PATCH 27/74] convert section on "package" term to note --- source/tutorials/packaging-existing-software.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/source/tutorials/packaging-existing-software.md b/source/tutorials/packaging-existing-software.md index 1bc41aa7f..1c6bac4cd 100644 --- a/source/tutorials/packaging-existing-software.md +++ b/source/tutorials/packaging-existing-software.md @@ -24,10 +24,11 @@ Then we'll move to progressively more complex packages with their own separate d Along the way, we'll encounter and address Nix error messages, build failures, and a host of other issues, developing our iterative debugging techniques as we go. -## Packages in Nix -Before we proceed, an important point of clarification: we conventionally use the term "package" by analogy to other systems, although this term is not a proper concept in Nix. +:::{note} +An important point of clarification: the term "package" is used conventionally by analogy to other systems, although the term does not refer to a proper concept in Nix. -For the purposes of this tutorial, by "package" we mean something like "result of a derivation"; this is the artifact you or others will use, as a consequence of having "packaged existing software with Nix". +For the purposes of this tutorial, "package" means something like "result of a derivation"; this is the artifact you or others will use, as a consequence of having "packaged existing software with Nix". +::: ## A Simple Project To start, we'll write a skeleton derivation, updating this as we go: From 388806960f0b2de8a25402a5c1ea2c490f099b91 Mon Sep 17 00:00:00 2001 From: Alexander Groleau Date: Mon, 7 Aug 2023 10:22:49 -0500 Subject: [PATCH 28/74] make skeleton derivation smaller --- source/tutorials/packaging-existing-software.md | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/source/tutorials/packaging-existing-software.md b/source/tutorials/packaging-existing-software.md index 1c6bac4cd..7c39424da 100644 --- a/source/tutorials/packaging-existing-software.md +++ b/source/tutorials/packaging-existing-software.md @@ -31,12 +31,10 @@ For the purposes of this tutorial, "package" means something like "result of a d ::: ## A Simple Project -To start, we'll write a skeleton derivation, updating this as we go: +To start, consider this skeleton derivation: ```nix -{ pkgs -, stdenv -}: +{ stdenv }: stdenv.mkDerivation { }; ``` @@ -45,6 +43,7 @@ stdenv.mkDerivation { }; Since GNU Hello is a popular (in a certain sense) package from the GNU Project, we can easily access its source code [from GNU's FTP](https://ftp.gnu.org/gnu/hello/). In this case, we'll download the [latest version](https://ftp.gnu.org/gnu/hello/hello-2.12.1.tar.gz) of `hello`, currently `2.12.1`. +This is a function which takes an attribute set containing `stdenv`, and produces a derivation (which currently does nothing). As you progress through this tutorial, you will update this several times, adding more details while following the general pattern. Downloading that into our build context is a good first step; we can do this in several ways, but it's best to use one of the nixpkgs builtin fetcher functions. From b315ef64fd3776bf5a2d4493e8322c17df3a896c Mon Sep 17 00:00:00 2001 From: Alexander Groleau Date: Mon, 7 Aug 2023 10:23:25 -0500 Subject: [PATCH 29/74] stop using royal we --- .../tutorials/packaging-existing-software.md | 112 ++++++++---------- 1 file changed, 52 insertions(+), 60 deletions(-) diff --git a/source/tutorials/packaging-existing-software.md b/source/tutorials/packaging-existing-software.md index 7c39424da..f38730961 100644 --- a/source/tutorials/packaging-existing-software.md +++ b/source/tutorials/packaging-existing-software.md @@ -15,14 +15,13 @@ One of Nix's primary use-cases is in addressing common difficulties encountered In the long term, Nix helps tremendously in alleviating that stress, but when *first* (re)packaging existing software with Nix, it's common to encounter missing dependencies preventing builds from succeeding. +In this tutorial, you'll create your first Nix derivations to package C/C++ software, taking advantage of the [`nixpkgs` `stdenv`](https://nixos.org/manual/nixpkgs/stable/#chap-stdenv) which automates much of the work of building self-contained C/C++ packages. -In this tutorial, we'll see how to create Nix derivations to package C/C++ software, taking advantage of the [`nixpkgs` `stdenv`](https://nixos.org/manual/nixpkgs/stable/#chap-stdenv) which automates much of the work of building self-contained C/C++ packages. +The tutorial begins by considering `hello`, an implementation of "hello world" which only requires dependencies already in `stdenv`. -We'll begin by considering `hello`, a feature-complete implementation of the famous "hello world", which requires no external dependencies. +Next, you will build more complex packages with their own dependencies, leading you to use additional derivation features. -Then we'll move to progressively more complex packages with their own separate dependencies, leading us to use additional derivation features. - -Along the way, we'll encounter and address Nix error messages, build failures, and a host of other issues, developing our iterative debugging techniques as we go. +You'll encounter and address Nix error messages, build failures, and a host of other issues, developing your iterative debugging techniques along the way. :::{note} An important point of clarification: the term "package" is used conventionally by analogy to other systems, although the term does not refer to a proper concept in Nix. @@ -39,17 +38,14 @@ To start, consider this skeleton derivation: stdenv.mkDerivation { }; ``` -### Hello, World! -Since GNU Hello is a popular (in a certain sense) package from the GNU Project, we can easily access its source code [from GNU's FTP](https://ftp.gnu.org/gnu/hello/). - -In this case, we'll download the [latest version](https://ftp.gnu.org/gnu/hello/hello-2.12.1.tar.gz) of `hello`, currently `2.12.1`. This is a function which takes an attribute set containing `stdenv`, and produces a derivation (which currently does nothing). As you progress through this tutorial, you will update this several times, adding more details while following the general pattern. -Downloading that into our build context is a good first step; we can do this in several ways, but it's best to use one of the nixpkgs builtin fetcher functions. +### Hello, World! +GNU Hello is an implementation of the "hello world" program, with source code accessible [from the GNU Project's FTP server](https://ftp.gnu.org/gnu/hello/). -In this case, we'll use `fetchTarball`, which takes the URI path to the download file and a SHA256 hash of its contents. +To begin, you will download the [latest version](https://ftp.gnu.org/gnu/hello/hello-2.12.1.tar.gz) of `hello` using `fetchTarball`, which takes the URI path to the download file and a SHA256 hash of its contents. -Here is our first iterative debugging technique: we can't actually know the hash until after we've downloaded and unpacked the tarball, but Nix will complain at us if the hash we supplied was incorrect, so we can just supply a fake one with `lib.fakeSha256` and change our derivation after Nix informs us of the correct hash: +The hash cannot be known until after the tarball has been downloaded and unpacked, but Nix will complain if the hash supplied to `fetchTarball` was incorrect, so it is common practice to supply a fake one with `lib.fakeSha256` and change the derivation definition after Nix reports the correct hash: ```nix # hello.nix @@ -66,7 +62,7 @@ stdenv.mkDerivation { } ``` -Let's save this file to `hello.nix` and try to build it. To do so, we'll use `nix-build`... +Save this file to `hello.nix` and try to build it with `nix-build`, observing your first build failure: ```console $ nix-build hello.nix @@ -84,12 +80,10 @@ error: cannot evaluate a function that has an argument without a value ('pkgs') 3| , lib ``` -... and immediately run into a problem: every derivation is a *function*, and functions need *arguments*! +Problem: the expression in `hello.nix` is a *function*, which only produces its intended output if it is passed the correct *arguments*. -### Your New Favorite Command -In order to pass the `pkgs` argument to our derivation, we'll need to import `nixpkgs` in another Nix expression. The `nix-build` command lets us pass whole expressions as an argument following the `-E/--expr` flag. - -We'll pass the following expression to `nix-build`: +### A New Command +In order to pass the `pkgs` argument to this derivation, you need to import `nixpkgs` with another Nix expression. The `nix-build` command allows passing whole expressions as an argument following the `-E/--expr` flag, like this one: ```console with import {}; callPackage ./hello.nix {} @@ -97,19 +91,19 @@ with import {}; callPackage ./hello.nix {} `callPackage` automatically passes attributes from `pkgs` to the given function, if they match attributes required by that function's argument attrset. Here, `callPackage` will supply `pkgs`, `lib`, and `stdenv`. - -Let's run this now: +Now run the full `nix-build` command with the new expression argument: ```console $ nix-build -E 'with import {}; callPackage ./hello.nix {}' error: derivation name missing ``` -Progress! The new failure occurs with the *derivation*, further down in the file than the initial error on line 2 about the `pkgs` argument not having a value; we successfully resolved the previous error by changing the expression passed to `nix-build`. + +This new failure occurs with the *derivation*, further down in the file than the initial error on line 2 about the `pkgs` argument not having a value; the previous error was successfully resolved by changing the expression passed to `nix-build`. ### Naming a Derivation Every derivation needs a `name` attribute, which must either be set directly or constructed by `mkDerivation` from `pname` and `version` attributes, if they exist. -Let's update the file again to add a `name`: +Update the file again to add a `name`: ```nix { pkgs @@ -153,7 +147,7 @@ error: ``` ### Finding The File Hash -As expected, Nix complained at us for lying about the file hash, and helpfully provided the correct one. We can substitute this into our `hello.nix` file, replacing `lib.fakeSha256`: +As expected, the incorrect file hash caused an error, and Nix helpfully provided the correct one, which you can now substitute into `hello.nix` to replace `lib.fakeSha256`: ```nix # hello.nix @@ -172,7 +166,7 @@ stdenv.mkDerivation { } ``` -Now let's run that command again: +Now run the previous command again: ```console $ nix-build -E 'with import {}; callPackage ./hello.nix {}' @@ -193,35 +187,33 @@ building build flags: SHELL=/nix/store/7q1b1bsmxi91zci6g8714rcljl620y7f-bash-5.2-p15/bin/bash ... ``` -Great news: our derivation built successfully! +Great news: the derivation built successfully! -We can see from the console output that `configure` was called, which produced a `Makefile` that was then used to build the project; we didn't actually write any build instructions, so we can surmise that Nix automatically detected the structure of the project directory. Indeed, the build system in `stdenv` is based on `autoconf`. +The console output shows that `configure` was called, which produced a `Makefile` that was then used to build the project; it wasn't necessary to write any build instructions in this case, because the `stdenv` build system is based on `autoconf`, which automatically detected the structure of the project directory. ### Build Result -We can check our working directory for the result: +Check your working directory for the result: ```console $ ls hello.nix result ``` -This result is a symbolic link to a Nix store location containing the built binary; we can call `./result/bin/hello` to execute this program: +This result is a symbolic link to a Nix store location containing the built binary; you can call `./result/bin/hello` to execute this program: ```console $ ./result/bin/hello Hello, world! ``` -We've successfully packaged our first program with Nix! +Congratulations, you have successfully packaged your first program with Nix! -Up next, we'll package another piece of software which has external dependencies that present new challenges, requiring us to lean more on `mkDerivation`. +Next, you'll package another piece of software with external-to-`stdenv` dependencies that present new challenges, requiring you to make use of more `mkDerivation` features. ## Something Bigger -The `hello` program is a simple and common place to start packaging, but it's not very useful or interesting, so we can't stop there. - -Now, we'll look at packaging a somewhat more complicated program, `icat`, which allows us to render images in our terminal. +Now you will package a somewhat more complicated program, `icat`, which allows you to render images in your terminal. -We'll start by copying `hello.nix` from the previous section to a new file, `icat.nix`. Then we'll make several changes, starting with the `name` attribute: +Start by copying `hello.nix` from the previous section to a new file, `icat.nix`, then update the `name` attribute in that file: ```nix # icat.nix @@ -239,7 +231,7 @@ stdenv.mkDerivation { } ``` -Now we'll download the source code. `icat`'s upstream repository is hosted on [GitHub](https://github.com/atextor/icat), so we should slightly modify our previous [source fetcher](https://nixos.org/manual/nixpkgs/stable/#chap-pkgs-fetchers): instead of `builtins.fetchTarball`, we'll use `pkgs.fetchFromGitHub`: +Now to download the source code; `icat`'s upstream repository is hosted on [GitHub](https://github.com/atextor/icat), so you should slightly modify the previous [source fetcher](https://nixos.org/manual/nixpkgs/stable/#chap-pkgs-fetchers): instead of `builtins.fetchTarball`, use `pkgs.fetchFromGitHub`: ```nix # icat.nix @@ -258,15 +250,15 @@ stdenv.mkDerivation { ``` ### Fetching Source from GitHub -While `fetchTarball` required `url` and `sha256` arguments, we'll need more than that for [`fetchFromGitHub`](https://nixos.org/manual/nixpkgs/stable/#fetchfromgithub). +While `fetchTarball` required `url` and `sha256` arguments, more are needed for [`fetchFromGitHub`](https://nixos.org/manual/nixpkgs/stable/#fetchfromgithub). -The source we want is hosted on GitHub at `https://github.com/atextor/icat`, which already gives us the first two arguments: +The source is hosted on GitHub at `https://github.com/atextor/icat`, which already gives the first two arguments: - `owner`: the name of the account controlling the repository; `owner = "atextor"` -- `repo`: the name of the repository we want to fetch; `repo = "icat"` +- `repo`: the name of the repository to fetch; `repo = "icat"` -We can navigate to the project's [Releases page](https://github.com/atextor/icat/releases) to find a suitable `rev`, such as the git commit hash or tag (e.g. `v1.0`) corresponding to the release we want to fetch. In this case, the latest release tag is `v0.5`. +You can navigate to the project's [Releases page](https://github.com/atextor/icat/releases) to find a suitable `rev`, such as the git commit hash or tag (e.g. `v1.0`) corresponding to the release you want to fetch. In this case, the latest release tag is `v0.5`. -As in the `hello` example, we also need to supply a hash. This time, instead of using `lib.fakeSha256` and letting `nix-build` report the correct one in an error, we'll fetch the correct hash in the first place with the `nix-prefetch-url` command. We want the SHA256 hash of the *contents* of the tarball, so we need to pass the `--unpack` and `--type sha256` arguments too: +As in the `hello` example, a hash must also be supplied. This time, instead of using `lib.fakeSha256` and letting `nix-build` report the correct one in an error, you can fetch the correct hash in the first place with the `nix-prefetch-url` command. You need the SHA256 hash of the *contents* of the tarball, so you will need to pass the `--unpack` and `--type sha256` arguments too: ```console $ nix-prefetch-url --unpack https://github.com/atextor/icat/archive/refs/tags/v0.5.tar.gz --type sha256 @@ -274,7 +266,7 @@ path is '/nix/store/p8jl1jlqxcsc7ryiazbpm7c1mqb6848b-v0.5.tar.gz' 0wyy2ksxp95vnh71ybj1bbmqd5ggp13x3mk37pzr99ljs9awy8ka ``` -Now we can supply the correct hash to `fetchFromGitHub`: +Now you can supply the correct hash to `fetchFromGitHub`: ```nix # icat.nix @@ -296,7 +288,7 @@ stdenv.mkDerivation { ``` ### Missing Dependencies -Running our previous `nix-build` invocation, we run into an entirely new issue: +Running the previous `nix-build` invocation, an entirely new issue is reported: ```console $ nix-build -E 'with import {}; callPackage ./icat.nix {}' @@ -338,13 +330,13 @@ error: builder for '/nix/store/al2wld63c66p3ln0rxqlkqqrqpspnicj-icat.drv' failed For full logs, run 'nix log /nix/store/al2wld63c66p3ln0rxqlkqqrqpspnicj-icat.drv'. ``` -Finally, a compiler error! We've successfully pulled the `icat` source from GitHub, and Nix tried to build what it found, but compilation failed due to a missing dependency: the `imlib2` header. If we [search for `imlib2` on search.nixos.org](https://search.nixos.org/packages?channel=23.05&from=0&size=50&sort=relevance&type=packages&query=imlib2), we'll find that `imlib2` is already in `nixpkgs`. +A compiler error! The `icat` source was pulled from GitHub, and Nix tried to build what it found, but compilation failed due to a missing dependency: the `imlib2` header. If you [search for `imlib2` on search.nixos.org](https://search.nixos.org/packages?channel=23.05&from=0&size=50&sort=relevance&type=packages&query=imlib2), you'll find that `imlib2` is already in `nixpkgs`. -We can add this package to our build environment by either +You can add this package to your build environment by either - adding `imlib2` to the set of inputs to the expression in `icat.nix`, and then adding `imlib2` to the list of `buildInputs` in `stdenv.mkDerivation`, or - adding `pkgs.imlib2` to the `buildInputs` directly, since `pkgs` is already in-scope. -Because `callPackage` is used to provide all necessary inputs in `nixpkgs` as well as in our `nix-build` invocation, the first approach is the one currently favored, and we'll use it here: +Because `callPackage` is used to provide all necessary inputs in `nixpkgs` as well as in the `nix-build` invocation, the first approach is the one currently favored, and you should use it here: ```nix # icat.nix @@ -368,7 +360,7 @@ stdenv.mkDerivation { } ``` -Another error, but we get further this time: +Another error, but compilation proceeds further this time: ```console $ nix-build -E 'with import {}; callPackage ./icat.nix {}' @@ -411,7 +403,7 @@ error: builder for '/nix/store/qg9f6zf0vwmvhz1w5i1fy2pw0l3wiqi9-icat.drv' failed For full logs, run 'nix log /nix/store/qg9f6zf0vwmvhz1w5i1fy2pw0l3wiqi9-icat.drv'. ``` -We can see a few warnings which should be corrected in the upstream code, but the important bit for our purposes is `fatal error: X11/Xlib.h: No such file or directory`: we're missing another dependency. +You can see a few warnings which should be corrected in the upstream code, but the important bit for this tutorial is `fatal error: X11/Xlib.h: No such file or directory`: another dependency is missing. In addition to the widespread practice of prefixing a project name with `lib` to indicate the libraries of that project, in Nixpkgs it's also common to separate headers, libraries, binaries, and documentation into different output attributes of a given [derivation](https://nixos.org/manual/nix/stable/language/derivations.html). @@ -419,7 +411,7 @@ In addition to the widespread practice of prefixing a project name with `lib` to Determining from where to source a dependency is currently a somewhat-involved process: it helps to become familiar with searching the `nixpkgs` source for keywords, in addition to checking discussion platforms like [the official NixOS Discourse](https://discourse.nixos.org). ::: -We need the `Xlib.h` headers from the `X11` C package, the Nixpkgs derivation for which is `libX11`, available in the `xorg` package set. The `Xlib` headers in turn live in the `dev` output of `xorg.libX11`. We'll add this to our derivation's input attribute set and to `buildInputs`: +You will need the `Xlib.h` headers from the `X11` C package, the Nixpkgs derivation for which is `libX11`, available in the `xorg` package set. The `Xlib` headers in turn live in the `dev` output of `xorg.libX11`. Add this to your derivation's input attribute set and to `buildInputs`: ```nix # icat.nix @@ -445,12 +437,12 @@ stdenv.mkDerivation { ``` :::{note} -We only added the top-level `xorg` derivation to the input attrset, rather than the full `xorg.libX11.dev` which would cause a syntax error. Because Nix is lazily-evaluated, including the dependency this way doesn't actually include all of `xorg` into our build context. +Only add the top-level `xorg` derivation to the input attrset, rather than the full `xorg.libX11.dev`, as the latter would cause a syntax error. Because Nix is lazily-evaluated, including the dependency this way is safe to do and doesn't actually include all of `xorg` into the build context. ::: ### `buildInputs` and `nativeBuildInputs` -Running our favorite command again: +Run the last command again: ```console $ nix-build -E 'with import {}; callPackage ./icat.nix {}' this derivation will be built: @@ -494,14 +486,14 @@ error: builder for '/nix/store/p21p5zkbwg83dhmi0bn1yz5ka6phd47x-icat.drv' failed For full logs, run 'nix log /nix/store/p21p5zkbwg83dhmi0bn1yz5ka6phd47x-icat.drv'. ``` -We've solved the missing dependency, but have another problem: `make: *** No rule to make target 'install'. Stop.` +The missing dependency error is solved, but there is now another problem: `make: *** No rule to make target 'install'. Stop.` ### installPhase -The `stdenv` is automatically working with the `Makefile` that comes with `icat`: we can see in the console output that `configure` and `make` are executed without issue, so the `icat` binary is compiling successfully. The failure occurs when the `stdenv` attempts to run `make install`: the `Makefile` included in the project happens to lack an `install` target, and the `README` in the `icat` repository only mentions using `make` to build the tool, leaving the installation step up to us. +The `stdenv` is automatically working with the `Makefile` that comes with `icat`: you can see in the console output that `configure` and `make` are executed without issue, so the `icat` binary is compiling successfully. The failure occurs when the `stdenv` attempts to run `make install`: the `Makefile` included in the project happens to lack an `install` target, and the `README` in the `icat` repository only mentions using `make` to build the tool, leaving the installation step up to users. -To add this step to our derivation, we use the [`installPhase` attribute](https://nixos.org/manual/nixpkgs/stable/#ssec-install-phase), which contains a list of command strings to execute to perform the installation. +To add this step to your derivation, use the [`installPhase` attribute](https://nixos.org/manual/nixpkgs/stable/#ssec-install-phase), which contains a list of command strings to execute to perform the installation. -Because the `make` step completes successfully, the `icat` executable is available in the build directory, and we only need to copy it from there to the output directory. In Nix, this location is stored in the `$out` variable, accessible in the derivation's component scripts; we'll create a `bin` directory within that and copy our `icat` binary there: +Because the `make` step completes successfully, the `icat` executable is available in the build directory, and you only need to copy it from there to the output directory. In Nix, this location is stored in the `$out` variable, accessible in the derivation's component scripts; create a `bin` directory within that and copy the `icat` binary there: ```nix # icat.nix @@ -534,15 +526,15 @@ stdenv.mkDerivation { ### Phases and Hooks Nix package derivations are separated into [phases](https://nixos.org/manual/nixpkgs/unstable/#sec-stdenv-phases), each of which is intended to control some aspect of the build process. -We saw earlier how the `stdenv` expected the project's `Makefile` to have an `install` target, and failed when it didn't. To fix this, we defined a custom `installPhase`, containing instructions for copying the `icat` binary to the correct output location, in effect installing it. +You saw earlier how the `stdenv` expected the project's `Makefile` to have an `install` target, and failed when it didn't. To fix this, you defined a custom `installPhase`, containing instructions for copying the `icat` binary to the correct output location, in effect installing it. -Up to that point, the `stdenv` automatically determined the `buildPhase` information for our `icat` package. +Up to that point, the `stdenv` automatically determined the `buildPhase` information for the `icat` package. During derivation realisation, there are a number of shell functions ("hooks", in `nixpkgs`) which may execute in each derivation phase, which do things like set variables, source files, create directories, and so on. These are specific to each phase, and run both before and after that phase's execution, controlling the build environment and helping to prevent environment-modifying behavior defined within packages from creating sources of nondeterminism within and between Nix derivations. It's good practice when packaging software with Nix to include calls to these hooks in the derivation phases you define, even when you don't make direct use of them; this facilitates easy [overriding](https://nixos.org/manual/nixpkgs/stable/#chap-overrides) of specific parts of the derivation later, in addition to the previously-mentioned reproducibility benefits. -We should now adjust our `installPhase` to call the appropriate hooks: +You should now adjust your `installPhase` to call the appropriate hooks: ```nix # icat.nix @@ -551,16 +543,16 @@ We should now adjust our `installPhase` to call the appropriate hooks: runHook preInstall mkdir -p $out/bin cp icat $out/bin - runHook postInstall + runHook postInstall ''; ... ``` -Running our `nix-build` command once more will finally do what we want, and more safely than before; we can `ls` in the local directory to find a `result` symlink to a location in the Nix store: +Running the `nix-build` command once more will finally do what you want, and more safely than before; you can `ls` in the local directory to find a `result` symlink to a location in the Nix store: ```console $ ls hello.nix icat.nix result ``` -`result/bin/icat` is the executable we built previously. Success! +`result/bin/icat` is the executable built previously. Success! From ac3ab131fbc0680ac09aa4afbe1a7c12f7b1c75d Mon Sep 17 00:00:00 2001 From: Alexander Groleau Date: Mon, 7 Aug 2023 12:06:30 -0500 Subject: [PATCH 30/74] don't introduce pkgs argument too soon --- .../tutorials/packaging-existing-software.md | 25 ++++++++----------- 1 file changed, 11 insertions(+), 14 deletions(-) diff --git a/source/tutorials/packaging-existing-software.md b/source/tutorials/packaging-existing-software.md index f38730961..a1cfac6d3 100644 --- a/source/tutorials/packaging-existing-software.md +++ b/source/tutorials/packaging-existing-software.md @@ -49,8 +49,7 @@ The hash cannot be known until after the tarball has been downloaded and unpacke ```nix # hello.nix -{ pkgs -, lib +{ lib , stdenv }: @@ -66,7 +65,7 @@ Save this file to `hello.nix` and try to build it with `nix-build`, observing yo ```console $ nix-build hello.nix -error: cannot evaluate a function that has an argument without a value ('pkgs') +error: cannot evaluate a function that has an argument without a value ('lib') Nix attempted to evaluate a function as a top level expression; in this case it must have its arguments supplied either by default values, or passed explicitly with '--arg' or '--argstr'. See @@ -75,21 +74,21 @@ error: cannot evaluate a function that has an argument without a value ('pkgs') at /home/nix-user/hello.nix:2:3: 1| # hello.nix - 2| { pkgs + 2| { lib | ^ - 3| , lib + 3| , stdenv ``` Problem: the expression in `hello.nix` is a *function*, which only produces its intended output if it is passed the correct *arguments*. ### A New Command -In order to pass the `pkgs` argument to this derivation, you need to import `nixpkgs` with another Nix expression. The `nix-build` command allows passing whole expressions as an argument following the `-E/--expr` flag, like this one: +`lib` is available from `nixpkgs`, which must be imported with another Nix expression in order to pass it as an argument to this derivation. The `nix-build` command allows passing whole expressions as an argument following the `-E/--expr` flag, like this one: ```console with import {}; callPackage ./hello.nix {} ``` -`callPackage` automatically passes attributes from `pkgs` to the given function, if they match attributes required by that function's argument attrset. Here, `callPackage` will supply `pkgs`, `lib`, and `stdenv`. +`callPackage` automatically passes attributes from `nixpkgs` to the given function (here, the one in `hello.nix`), if they match attributes required by that function's argument attrset. Here, `callPackage` will supply `lib`, and `stdenv`. Now run the full `nix-build` command with the new expression argument: @@ -98,7 +97,7 @@ $ nix-build -E 'with import {}; callPackage ./hello.nix {}' error: derivation name missing ``` -This new failure occurs with the *derivation*, further down in the file than the initial error on line 2 about the `pkgs` argument not having a value; the previous error was successfully resolved by changing the expression passed to `nix-build`. +This new failure occurs with the *derivation*, further down in the file than the initial error on line 2 about the `lib` argument not having a value; the previous error was successfully resolved by changing the expression passed to `nix-build`. ### Naming a Derivation Every derivation needs a `name` attribute, which must either be set directly or constructed by `mkDerivation` from `pname` and `version` attributes, if they exist. @@ -106,8 +105,8 @@ Every derivation needs a `name` attribute, which must either be set directly or Update the file again to add a `name`: ```nix -{ pkgs -, lib +# hello.nix +{ lib , stdenv }: @@ -151,8 +150,7 @@ As expected, the incorrect file hash caused an error, and Nix helpfully provided ```nix # hello.nix -{ pkgs -, lib +{ lib , stdenv }: @@ -217,8 +215,7 @@ Start by copying `hello.nix` from the previous section to a new file, `icat.nix` ```nix # icat.nix -{ pkgs -, lib +{ lib , stdenv }: From 0b8f525a94998676f210285807f5adcf55eda030 Mon Sep 17 00:00:00 2001 From: Alexander Groleau Date: Mon, 7 Aug 2023 12:09:02 -0500 Subject: [PATCH 31/74] use default.nix workflow --- .../tutorials/packaging-existing-software.md | 71 ++++++++++++++----- 1 file changed, 52 insertions(+), 19 deletions(-) diff --git a/source/tutorials/packaging-existing-software.md b/source/tutorials/packaging-existing-software.md index a1cfac6d3..7f145e64d 100644 --- a/source/tutorials/packaging-existing-software.md +++ b/source/tutorials/packaging-existing-software.md @@ -90,10 +90,24 @@ with import {}; callPackage ./hello.nix {} `callPackage` automatically passes attributes from `nixpkgs` to the given function (here, the one in `hello.nix`), if they match attributes required by that function's argument attrset. Here, `callPackage` will supply `lib`, and `stdenv`. -Now run the full `nix-build` command with the new expression argument: +To avoid having to execute `nix-build -E 'with import {}; callPackage ./hello.nix {}'` each time, create a `default.nix` in the same directory as `hello.nix`, with the following contents: + +```nix +# default.nix +let + pkgs = import { }; +in +{ + hello = pkgs.callPackage ./hello.nix { }; +} +``` + +This allows you to use `nix-build -A hello` to realize the derivation in `hello.nix`, similar to the current convention used in `nixpkgs`. + +Now run the `nix-build` command with the new argument: ```console -$ nix-build -E 'with import {}; callPackage ./hello.nix {}' +$ nix-build -A hello error: derivation name missing ``` @@ -122,23 +136,23 @@ stdenv.mkDerivation { and then re-run the command: ```console -$ nix-build -E 'with import {}; callPackage ./hello.nix {}' +$ nix-build -A hello error: … while calling the 'derivationStrict' builtin at /builtin/derivation.nix:9:12: (source not available) … while evaluating derivation 'hello' - whose name attribute is located at /nix/store/i6w7hmdjp1jg71g7xbjgz5rn96q443c6-nixos-23.05.1471.b72aa95f7f0/nixos/pkgs/stdenv/generic/make-derivation.nix:303:7 + whose name attribute is located at /nix/store/5na8c1j0cn5wls2g2d8q357cz3aqlaqj-nixos-23.05.2478.bd836ac5e5a7/nixos/pkgs/stdenv/generic/make-derivation.nix:303:7 … while evaluating attribute 'src' of derivation 'hello' - at /home/nix-user/hello.nix:5:3: + at /home/nix-user/hello.nix:9:3: - 4| name = "hello"; - 5| src = builtins.fetchTarball { + 8| + 9| src = builtins.fetchTarball { | ^ - 6| url = "https://ftp.gnu.org/gnu/hello/hello-2.12.1.tar.gz"; + 10| url = "https://ftp.gnu.org/gnu/hello/hello-2.12.1.tar.gz"; error: hash mismatch in file downloaded from 'https://ftp.gnu.org/gnu/hello/hello-2.12.1.tar.gz': specified: sha256:0000000000000000000000000000000000000000000000000000 @@ -167,7 +181,7 @@ stdenv.mkDerivation { Now run the previous command again: ```console -$ nix-build -E 'with import {}; callPackage ./hello.nix {}' +$ nix-build -A hello this derivation will be built: /nix/store/rbq37s3r76rr77c7d8x8px7z04kw2mk7-hello.drv building '/nix/store/rbq37s3r76rr77c7d8x8px7z04kw2mk7-hello.drv'... @@ -194,7 +208,7 @@ Check your working directory for the result: ```console $ ls -hello.nix result +default.nix hello.nix result ``` This result is a symbolic link to a Nix store location containing the built binary; you can call `./result/bin/hello` to execute this program: @@ -211,7 +225,20 @@ Next, you'll package another piece of software with external-to-`stdenv` depende ## Something Bigger Now you will package a somewhat more complicated program, `icat`, which allows you to render images in your terminal. -Start by copying `hello.nix` from the previous section to a new file, `icat.nix`, then update the `name` attribute in that file: +To start, modify the `default.nix` from the previous section by adding a new attribute for `icat`: + +```nix +# default.nix +let + pkgs = import { }; +in +{ + hello = pkgs.callPackage ./hello.nix { }; + icat = pkgs.callPackage ./icat.nix { }; +} +``` + +Now copy `hello.nix` to a new file, `icat.nix`, and update the `name` attribute in that file: ```nix # icat.nix @@ -285,13 +312,19 @@ stdenv.mkDerivation { ``` ### Missing Dependencies -Running the previous `nix-build` invocation, an entirely new issue is reported: +Running `nix-build` with the new `icat` attribute, an entirely new issue is reported: ```console -$ nix-build -E 'with import {}; callPackage ./icat.nix {}' -this derivation will be built: - /nix/store/al2wld63c66p3ln0rxqlkqqrqpspnicj-icat.drv -building '/nix/store/al2wld63c66p3ln0rxqlkqqrqpspnicj-icat.drv'... +$ nix-build -A icat +these 2 derivations will be built: + /nix/store/86q9x927hsyyzfr4lcqirmsbimysi6mb-source.drv + /nix/store/l5wz9inkvkf0qhl8kpl39vpg2xfm2qpy-icat.drv +these 19 paths will be fetched (4.21 MiB download, 17.65 MiB unpacked): +... +trying https://github.com/atextor/icat/archive/v0.5.tar.gz +... +unpacking source archive /build/v0.5.tar.gz +building '/nix/store/l5wz9inkvkf0qhl8kpl39vpg2xfm2qpy-icat.drv'... unpacking sources unpacking source archive /nix/store/rx21f6fgnmxgp1sw0wbqll9wds4xc6v0-source source root is source @@ -360,7 +393,7 @@ stdenv.mkDerivation { Another error, but compilation proceeds further this time: ```console -$ nix-build -E 'with import {}; callPackage ./icat.nix {}' +$ nix-build -A icat this derivation will be built: /nix/store/qg9f6zf0vwmvhz1w5i1fy2pw0l3wiqi9-icat.drv building '/nix/store/qg9f6zf0vwmvhz1w5i1fy2pw0l3wiqi9-icat.drv'... @@ -441,7 +474,7 @@ Only add the top-level `xorg` derivation to the input attrset, rather than the f ### `buildInputs` and `nativeBuildInputs` Run the last command again: ```console -$ nix-build -E 'with import {}; callPackage ./icat.nix {}' +$ nix-build -A icat this derivation will be built: /nix/store/p21p5zkbwg83dhmi0bn1yz5ka6phd47x-icat.drv building '/nix/store/p21p5zkbwg83dhmi0bn1yz5ka6phd47x-icat.drv'... @@ -549,7 +582,7 @@ Running the `nix-build` command once more will finally do what you want, and mor ```console $ ls -hello.nix icat.nix result +default.nix hello.nix icat.nix result ``` `result/bin/icat` is the executable built previously. Success! From bbacf0d839c9e09f71f3367a7ccd82f0a558d6d7 Mon Sep 17 00:00:00 2001 From: Alexander Groleau Date: Mon, 7 Aug 2023 12:09:19 -0500 Subject: [PATCH 32/74] add pkgs argument again --- source/tutorials/packaging-existing-software.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/source/tutorials/packaging-existing-software.md b/source/tutorials/packaging-existing-software.md index 7f145e64d..a4f2347f8 100644 --- a/source/tutorials/packaging-existing-software.md +++ b/source/tutorials/packaging-existing-software.md @@ -255,7 +255,7 @@ stdenv.mkDerivation { } ``` -Now to download the source code; `icat`'s upstream repository is hosted on [GitHub](https://github.com/atextor/icat), so you should slightly modify the previous [source fetcher](https://nixos.org/manual/nixpkgs/stable/#chap-pkgs-fetchers): instead of `builtins.fetchTarball`, use `pkgs.fetchFromGitHub`: +Now to download the source code. `icat`'s upstream repository is hosted on [GitHub](https://github.com/atextor/icat), so you should slightly modify the previous [source fetcher](https://nixos.org/manual/nixpkgs/stable/#chap-pkgs-fetchers), this time using `pkgs.fetchFromGitHub` instead of `builtins.fetchTarball`, updating the argument attribute set to the function accordingly: ```nix # icat.nix From 475ec56b15702a6996bbccbc1a3825004dd2b6da Mon Sep 17 00:00:00 2001 From: Alexander Groleau Date: Mon, 7 Aug 2023 12:10:17 -0500 Subject: [PATCH 33/74] update console output --- .../tutorials/packaging-existing-software.md | 61 ++++++++++--------- 1 file changed, 31 insertions(+), 30 deletions(-) diff --git a/source/tutorials/packaging-existing-software.md b/source/tutorials/packaging-existing-software.md index a4f2347f8..dd41dad17 100644 --- a/source/tutorials/packaging-existing-software.md +++ b/source/tutorials/packaging-existing-software.md @@ -332,12 +332,12 @@ patching sources configuring no configure script, doing nothing building -build flags: SHELL=/nix/store/7q1b1bsmxi91zci6g8714rcljl620y7f-bash-5.2-p15/bin/bash +build flags: SHELL=/nix/store/8fv91097mbh5049i9rglc73dx6kjg3qk-bash-5.2-p15/bin/bash gcc -c -Wall -pedantic -std=c99 -D_BSD_SOURCE -o icat.o icat.c -In file included from /nix/store/dpk5m64n0axk01fq8h2m0yl9hhpq2nqk-glibc-2.37-8-dev/include/bits/libc-header-start.h:33, - from /nix/store/dpk5m64n0axk01fq8h2m0yl9hhpq2nqk-glibc-2.37-8-dev/include/stdio.h:27, +In file included from /nix/store/hkj250rjsvxcbr31fr1v81cv88cdfp4l-glibc-2.37-8-dev/include/bits/libc-header-start.h:33, + from /nix/store/hkj250rjsvxcbr31fr1v81cv88cdfp4l-glibc-2.37-8-dev/include/stdio.h:27, from icat.c:31: -/nix/store/dpk5m64n0axk01fq8h2m0yl9hhpq2nqk-glibc-2.37-8-dev/include/features.h:195:3: warning: #warning "_BSD_SOURCE and _SVID_SOURCE are deprecated, use _DEFAULT_SOURCE" [8;;https://gcc.gnu.org/onlinedocs/gcc/Warning-Options.html#index-Wcpp-Wcpp8;;] +/nix/store/hkj250rjsvxcbr31fr1v81cv88cdfp4l-glibc-2.37-8-dev/include/features.h:195:3: warning: #warning "_BSD_SOURCE and _SVID_SOURCE are deprecated, use _DEFAULT_SOURCE" [8;;https://gcc.gnu.org/onlinedocs/gcc/Warning-Options.html#index-Wcpp-Wcpp8;;] 195 | # warning "_BSD_SOURCE and _SVID_SOURCE are deprecated, use _DEFAULT_SOURCE" | ^~~~~~~ icat.c:39:10: fatal error: Imlib2.h: No such file or directory @@ -345,11 +345,11 @@ icat.c:39:10: fatal error: Imlib2.h: No such file or directory | ^~~~~~~~~~ compilation terminated. make: *** [Makefile:16: icat.o] Error 1 -error: builder for '/nix/store/al2wld63c66p3ln0rxqlkqqrqpspnicj-icat.drv' failed with exit code 2; +error: builder for '/nix/store/l5wz9inkvkf0qhl8kpl39vpg2xfm2qpy-icat.drv' failed with exit code 2; last 10 log lines: - > from /nix/store/dpk5m64n0axk01fq8h2m0yl9hhpq2nqk-glibc-2.37-8-dev/include/stdio.h:27, + > from /nix/store/hkj250rjsvxcbr31fr1v81cv88cdfp4l-glibc-2.37-8-dev/include/stdio.h:27, > from icat.c:31: - > /nix/store/dpk5m64n0axk01fq8h2m0yl9hhpq2nqk-glibc-2.37-8-dev/include/features.h:195:3: warning: #warning "_BSD_SOURCE and _SVID_SOURCE are deprecated, use _DEFAULT_SOURCE" [8;;https://gcc.gnu.org/onlinedocs/gcc/Warning-Options.html#index-Wcpp-Wcpp8;;] + > /nix/store/hkj250rjsvxcbr31fr1v81cv88cdfp4l-glibc-2.37-8-dev/include/features.h:195:3: warning: #warning "_BSD_SOURCE and _SVID_SOURCE are deprecated, use _DEFAULT_SOURCE" [8;;https://gcc.gnu.org/onlinedocs/gcc/Warning-Options.html#index-Wcpp-Wcpp8;;] > 195 | # warning "_BSD_SOURCE and _SVID_SOURCE are deprecated, use _DEFAULT_SOURCE" > | ^~~~~~~ > icat.c:39:10: fatal error: Imlib2.h: No such file or directory @@ -357,7 +357,7 @@ error: builder for '/nix/store/al2wld63c66p3ln0rxqlkqqrqpspnicj-icat.drv' failed > | ^~~~~~~~~~ > compilation terminated. > make: *** [Makefile:16: icat.o] Error 1 - For full logs, run 'nix log /nix/store/al2wld63c66p3ln0rxqlkqqrqpspnicj-icat.drv'. + For full logs, run 'nix log /nix/store/l5wz9inkvkf0qhl8kpl39vpg2xfm2qpy-icat.drv'. ``` A compiler error! The `icat` source was pulled from GitHub, and Nix tried to build what it found, but compilation failed due to a missing dependency: the `imlib2` header. If you [search for `imlib2` on search.nixos.org](https://search.nixos.org/packages?channel=23.05&from=0&size=50&sort=relevance&type=packages&query=imlib2), you'll find that `imlib2` is already in `nixpkgs`. @@ -395,8 +395,9 @@ Another error, but compilation proceeds further this time: ```console $ nix-build -A icat this derivation will be built: - /nix/store/qg9f6zf0vwmvhz1w5i1fy2pw0l3wiqi9-icat.drv -building '/nix/store/qg9f6zf0vwmvhz1w5i1fy2pw0l3wiqi9-icat.drv'... + /nix/store/bw2d4rp2k1l5rg49hds199ma2mz36x47-icat.drv +... +building '/nix/store/bw2d4rp2k1l5rg49hds199ma2mz36x47-icat.drv'... unpacking sources unpacking source archive /nix/store/rx21f6fgnmxgp1sw0wbqll9wds4xc6v0-source source root is source @@ -404,33 +405,33 @@ patching sources configuring no configure script, doing nothing building -build flags: SHELL=/nix/store/7q1b1bsmxi91zci6g8714rcljl620y7f-bash-5.2-p15/bin/bash +build flags: SHELL=/nix/store/8fv91097mbh5049i9rglc73dx6kjg3qk-bash-5.2-p15/bin/bash gcc -c -Wall -pedantic -std=c99 -D_BSD_SOURCE -o icat.o icat.c -In file included from /nix/store/dpk5m64n0axk01fq8h2m0yl9hhpq2nqk-glibc-2.37-8-dev/include/bits/libc-header-start.h:33, - from /nix/store/dpk5m64n0axk01fq8h2m0yl9hhpq2nqk-glibc-2.37-8-dev/include/stdio.h:27, +In file included from /nix/store/hkj250rjsvxcbr31fr1v81cv88cdfp4l-glibc-2.37-8-dev/include/bits/libc-header-start.h:33, + from /nix/store/hkj250rjsvxcbr31fr1v81cv88cdfp4l-glibc-2.37-8-dev/include/stdio.h:27, from icat.c:31: -/nix/store/dpk5m64n0axk01fq8h2m0yl9hhpq2nqk-glibc-2.37-8-dev/include/features.h:195:3: warning: #warning "_BSD_SOURCE and _SVID_SOURCE are deprecated, use _DEFAULT_SOURCE" [8;;https://gcc.gnu.org/onlinedocs/gcc/Warning-Options.html#index-Wcpp-Wcpp8;;] +/nix/store/hkj250rjsvxcbr31fr1v81cv88cdfp4l-glibc-2.37-8-dev/include/features.h:195:3: warning: #warning "_BSD_SOURCE and _SVID_SOURCE are deprecated, use _DEFAULT_SOURCE" [8;;https://gcc.gnu.org/onlinedocs/gcc/Warning-Options.html#index-Wcpp-Wcpp8;;] 195 | # warning "_BSD_SOURCE and _SVID_SOURCE are deprecated, use _DEFAULT_SOURCE" | ^~~~~~~ In file included from icat.c:39: -/nix/store/hkgbjcr182m3q9xs0j1qmp3dh08mbg31-imlib2-1.11.1-dev/include/Imlib2.h:45:10: fatal error: X11/Xlib.h: No such file or directory +/nix/store/4fvrh0sjc8sbkbqda7dfsh7q0gxmnh9p-imlib2-1.11.1-dev/include/Imlib2.h:45:10: fatal error: X11/Xlib.h: No such file or directory 45 | #include | ^~~~~~~~~~~~ compilation terminated. make: *** [Makefile:16: icat.o] Error 1 -error: builder for '/nix/store/qg9f6zf0vwmvhz1w5i1fy2pw0l3wiqi9-icat.drv' failed with exit code 2; +error: builder for '/nix/store/bw2d4rp2k1l5rg49hds199ma2mz36x47-icat.drv' failed with exit code 2; last 10 log lines: > from icat.c:31: - > /nix/store/dpk5m64n0axk01fq8h2m0yl9hhpq2nqk-glibc-2.37-8-dev/include/features.h:195:3: warning: #warning "_BSD_SOURCE and _SVID_SOURCE are deprecated, use _DEFAULT_SOURCE" [8;;https://gcc.gnu.org/onlinedocs/gcc/Warning-Options.html#index-Wcpp-Wcpp8;;] + > /nix/store/hkj250rjsvxcbr31fr1v81cv88cdfp4l-glibc-2.37-8-dev/include/features.h:195:3: warning: #warning "_BSD_SOURCE and _SVID_SOURCE are deprecated, use _DEFAULT_SOURCE" [8;;https://gcc.gnu.org/onlinedocs/gcc/Warning-Options.html#index-Wcpp-Wcpp8;;] > 195 | # warning "_BSD_SOURCE and _SVID_SOURCE are deprecated, use _DEFAULT_SOURCE" > | ^~~~~~~ > In file included from icat.c:39: - > /nix/store/hkgbjcr182m3q9xs0j1qmp3dh08mbg31-imlib2-1.11.1-dev/include/Imlib2.h:45:10: fatal error: X11/Xlib.h: No such file or directory + > /nix/store/4fvrh0sjc8sbkbqda7dfsh7q0gxmnh9p-imlib2-1.11.1-dev/include/Imlib2.h:45:10: fatal error: X11/Xlib.h: No such file or directory > 45 | #include > | ^~~~~~~~~~~~ > compilation terminated. > make: *** [Makefile:16: icat.o] Error 1 - For full logs, run 'nix log /nix/store/qg9f6zf0vwmvhz1w5i1fy2pw0l3wiqi9-icat.drv'. + For full logs, run 'nix log /nix/store/bw2d4rp2k1l5rg49hds199ma2mz36x47-icat.drv'. ``` You can see a few warnings which should be corrected in the upstream code, but the important bit for this tutorial is `fatal error: X11/Xlib.h: No such file or directory`: another dependency is missing. @@ -476,8 +477,8 @@ Run the last command again: ```console $ nix-build -A icat this derivation will be built: - /nix/store/p21p5zkbwg83dhmi0bn1yz5ka6phd47x-icat.drv -building '/nix/store/p21p5zkbwg83dhmi0bn1yz5ka6phd47x-icat.drv'... + /nix/store/x1d79ld8jxqdla5zw2b47d2sl87mf56k-icat.drv +building '/nix/store/x1d79ld8jxqdla5zw2b47d2sl87mf56k-icat.drv'... unpacking sources unpacking source archive /nix/store/rx21f6fgnmxgp1sw0wbqll9wds4xc6v0-source source root is source @@ -485,12 +486,12 @@ patching sources configuring no configure script, doing nothing building -build flags: SHELL=/nix/store/7q1b1bsmxi91zci6g8714rcljl620y7f-bash-5.2-p15/bin/bash +build flags: SHELL=/nix/store/8fv91097mbh5049i9rglc73dx6kjg3qk-bash-5.2-p15/bin/bash gcc -c -Wall -pedantic -std=c99 -D_BSD_SOURCE -o icat.o icat.c -In file included from /nix/store/dpk5m64n0axk01fq8h2m0yl9hhpq2nqk-glibc-2.37-8-dev/include/bits/libc-header-start.h:33, - from /nix/store/dpk5m64n0axk01fq8h2m0yl9hhpq2nqk-glibc-2.37-8-dev/include/stdio.h:27, +In file included from /nix/store/hkj250rjsvxcbr31fr1v81cv88cdfp4l-glibc-2.37-8-dev/include/bits/libc-header-start.h:33, + from /nix/store/hkj250rjsvxcbr31fr1v81cv88cdfp4l-glibc-2.37-8-dev/include/stdio.h:27, from icat.c:31: -/nix/store/dpk5m64n0axk01fq8h2m0yl9hhpq2nqk-glibc-2.37-8-dev/include/features.h:195:3: warning: #warning "_BSD_SOURCE and _SVID_SOURCE are deprecated, use _DEFAULT_SOURCE" [8;;https://gcc.gnu.org/onlinedocs/gcc/Warning-Options.html#index-Wcpp-Wcpp8;;] +/nix/store/hkj250rjsvxcbr31fr1v81cv88cdfp4l-glibc-2.37-8-dev/include/features.h:195:3: warning: #warning "_BSD_SOURCE and _SVID_SOURCE are deprecated, use _DEFAULT_SOURCE" [8;;https://gcc.gnu.org/onlinedocs/gcc/Warning-Options.html#index-Wcpp-Wcpp8;;] 195 | # warning "_BSD_SOURCE and _SVID_SOURCE are deprecated, use _DEFAULT_SOURCE" | ^~~~~~~ icat.c: In function 'main': @@ -499,9 +500,9 @@ icat.c:319:33: warning: ignoring return value of 'write' declared with attribute | ^~~~~~~~~~~~~~~~~~~~~~~~ gcc -o icat icat.o -lImlib2 installing -install flags: SHELL=/nix/store/7q1b1bsmxi91zci6g8714rcljl620y7f-bash-5.2-p15/bin/bash install +install flags: SHELL=/nix/store/8fv91097mbh5049i9rglc73dx6kjg3qk-bash-5.2-p15/bin/bash install make: *** No rule to make target 'install'. Stop. -error: builder for '/nix/store/p21p5zkbwg83dhmi0bn1yz5ka6phd47x-icat.drv' failed with exit code 2; +error: builder for '/nix/store/x1d79ld8jxqdla5zw2b47d2sl87mf56k-icat.drv' failed with exit code 2; last 10 log lines: > 195 | # warning "_BSD_SOURCE and _SVID_SOURCE are deprecated, use _DEFAULT_SOURCE" > | ^~~~~~~ @@ -511,9 +512,9 @@ error: builder for '/nix/store/p21p5zkbwg83dhmi0bn1yz5ka6phd47x-icat.drv' failed > | ^~~~~~~~~~~~~~~~~~~~~~~~ > gcc -o icat icat.o -lImlib2 > installing - > install flags: SHELL=/nix/store/7q1b1bsmxi91zci6g8714rcljl620y7f-bash-5.2-p15/bin/bash install + > install flags: SHELL=/nix/store/8fv91097mbh5049i9rglc73dx6kjg3qk-bash-5.2-p15/bin/bash install > make: *** No rule to make target 'install'. Stop. - For full logs, run 'nix log /nix/store/p21p5zkbwg83dhmi0bn1yz5ka6phd47x-icat.drv'. + For full logs, run 'nix log /nix/store/x1d79ld8jxqdla5zw2b47d2sl87mf56k-icat.drv'. ``` The missing dependency error is solved, but there is now another problem: `make: *** No rule to make target 'install'. Stop.` @@ -544,7 +545,7 @@ stdenv.mkDerivation { sha256 = "0wyy2ksxp95vnh71ybj1bbmqd5ggp13x3mk37pzr99ljs9awy8ka"; }; - buildInputs = with pkgs; [ imlib2 xorg.libX11.dev ]; + buildInputs = [ imlib2 xorg.libX11.dev ]; installPhase = '' mkdir -p $out/bin From a7a245be62158d3db078516b76873f911925bc26 Mon Sep 17 00:00:00 2001 From: Alexander Groleau Date: Mon, 7 Aug 2023 12:10:30 -0500 Subject: [PATCH 34/74] only use 4 spaces, no tabs --- source/tutorials/packaging-existing-software.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/source/tutorials/packaging-existing-software.md b/source/tutorials/packaging-existing-software.md index dd41dad17..28e8cf13f 100644 --- a/source/tutorials/packaging-existing-software.md +++ b/source/tutorials/packaging-existing-software.md @@ -460,7 +460,7 @@ stdenv.mkDerivation { owner = "atextor"; repo = "icat"; rev = "v0.5"; - sha256 = "0wyy2ksxp95vnh71ybj1bbmqd5ggp13x3mk37pzr99ljs9awy8ka"; + sha256 = "0wyy2ksxp95vnh71ybj1bbmqd5ggp13x3mk37pzr99ljs9awy8ka"; }; buildInputs = [ imlib2 xorg.libX11.dev ]; From 34d55f8ecc0ee429a5afc27f1505c27f6a284e79 Mon Sep 17 00:00:00 2001 From: Alexander Groleau Date: Thu, 10 Aug 2023 10:43:51 -0500 Subject: [PATCH 35/74] update learning journey index --- source/tutorials/learning-journey/index.md | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/source/tutorials/learning-journey/index.md b/source/tutorials/learning-journey/index.md index d7778ea18..80fa8b327 100644 --- a/source/tutorials/learning-journey/index.md +++ b/source/tutorials/learning-journey/index.md @@ -1,10 +1,12 @@ +(learning-journey)= # Learning Journey This collection of tutorials guides you through your first steps with Nix. This series is a work in progress and will have some overlap with existing tutorials. The intention is to unify these tutorials over time. - ```{toctree} - :maxdepth: 1 - shell-dot-nix.md - ``` +```{toctree} +:maxdepth: 1 +packaging-existing-software.md +shell-dot-nix.md +``` From d73c0c7979a68fe39b54a1ae485bf8af84b89d48 Mon Sep 17 00:00:00 2001 From: Alexander Groleau Date: Thu, 10 Aug 2023 10:44:50 -0500 Subject: [PATCH 36/74] move packaging existing software to learning journey --- source/tutorials/index.md | 1 - .../{ => learning-journey}/packaging-existing-software.md | 0 2 files changed, 1 deletion(-) rename source/tutorials/{ => learning-journey}/packaging-existing-software.md (100%) diff --git a/source/tutorials/index.md b/source/tutorials/index.md index 41fe9d838..a5f93a1bc 100644 --- a/source/tutorials/index.md +++ b/source/tutorials/index.md @@ -12,5 +12,4 @@ first-steps/index.md learning-journey/index.md nixos/index.md cross-compilation.md -packaging-existing-software.md ``` diff --git a/source/tutorials/packaging-existing-software.md b/source/tutorials/learning-journey/packaging-existing-software.md similarity index 100% rename from source/tutorials/packaging-existing-software.md rename to source/tutorials/learning-journey/packaging-existing-software.md From 554aeec5cb59d718958d437e80e38441c5e9bcdd Mon Sep 17 00:00:00 2001 From: Alexander Groleau Date: Tue, 8 Aug 2023 10:48:50 -0500 Subject: [PATCH 37/74] Update source/tutorials/learning-journey/packaging-existing-software.md Co-authored-by: Silvan Mosberger --- .../tutorials/learning-journey/packaging-existing-software.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/source/tutorials/learning-journey/packaging-existing-software.md b/source/tutorials/learning-journey/packaging-existing-software.md index 28e8cf13f..47e235968 100644 --- a/source/tutorials/learning-journey/packaging-existing-software.md +++ b/source/tutorials/learning-journey/packaging-existing-software.md @@ -223,7 +223,7 @@ Congratulations, you have successfully packaged your first program with Nix! Next, you'll package another piece of software with external-to-`stdenv` dependencies that present new challenges, requiring you to make use of more `mkDerivation` features. ## Something Bigger -Now you will package a somewhat more complicated program, `icat`, which allows you to render images in your terminal. +Now you will package a somewhat more complicated program, [`icat`](https://github.com/atextor/icat), which allows you to render images in your terminal. To start, modify the `default.nix` from the previous section by adding a new attribute for `icat`: From beef4414abf843806e89ccaea9af206ad2efcadb Mon Sep 17 00:00:00 2001 From: Alexander Groleau Date: Tue, 8 Aug 2023 10:49:51 -0500 Subject: [PATCH 38/74] Update source/tutorials/learning-journey/packaging-existing-software.md Co-authored-by: Silvan Mosberger --- .../tutorials/learning-journey/packaging-existing-software.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/source/tutorials/learning-journey/packaging-existing-software.md b/source/tutorials/learning-journey/packaging-existing-software.md index 47e235968..c6e5f48bc 100644 --- a/source/tutorials/learning-journey/packaging-existing-software.md +++ b/source/tutorials/learning-journey/packaging-existing-software.md @@ -282,7 +282,7 @@ The source is hosted on GitHub at `https://github.com/atextor/icat`, which alrea You can navigate to the project's [Releases page](https://github.com/atextor/icat/releases) to find a suitable `rev`, such as the git commit hash or tag (e.g. `v1.0`) corresponding to the release you want to fetch. In this case, the latest release tag is `v0.5`. -As in the `hello` example, a hash must also be supplied. This time, instead of using `lib.fakeSha256` and letting `nix-build` report the correct one in an error, you can fetch the correct hash in the first place with the `nix-prefetch-url` command. You need the SHA256 hash of the *contents* of the tarball, so you will need to pass the `--unpack` and `--type sha256` arguments too: +As in the `hello` example, a hash must also be supplied. This time, instead of using `lib.fakeSha256` and letting `nix-build` report the correct one in an error, you can fetch the correct hash in the first place with the `nix-prefetch-url` command. You need the SHA256 hash of the *contents* of the tarball (as opposed to the hash of the tarball file itself), so you will need to pass the `--unpack` and `--type sha256` arguments too: ```console $ nix-prefetch-url --unpack https://github.com/atextor/icat/archive/refs/tags/v0.5.tar.gz --type sha256 From 5184c898243ab846538c8f4f0ec19dc412e5da95 Mon Sep 17 00:00:00 2001 From: Alexander Groleau Date: Tue, 8 Aug 2023 10:50:27 -0500 Subject: [PATCH 39/74] Update source/tutorials/learning-journey/packaging-existing-software.md Co-authored-by: Silvan Mosberger --- .../learning-journey/packaging-existing-software.md | 6 +----- 1 file changed, 1 insertion(+), 5 deletions(-) diff --git a/source/tutorials/learning-journey/packaging-existing-software.md b/source/tutorials/learning-journey/packaging-existing-software.md index c6e5f48bc..657ef1abd 100644 --- a/source/tutorials/learning-journey/packaging-existing-software.md +++ b/source/tutorials/learning-journey/packaging-existing-software.md @@ -362,11 +362,7 @@ error: builder for '/nix/store/l5wz9inkvkf0qhl8kpl39vpg2xfm2qpy-icat.drv' failed A compiler error! The `icat` source was pulled from GitHub, and Nix tried to build what it found, but compilation failed due to a missing dependency: the `imlib2` header. If you [search for `imlib2` on search.nixos.org](https://search.nixos.org/packages?channel=23.05&from=0&size=50&sort=relevance&type=packages&query=imlib2), you'll find that `imlib2` is already in `nixpkgs`. -You can add this package to your build environment by either -- adding `imlib2` to the set of inputs to the expression in `icat.nix`, and then adding `imlib2` to the list of `buildInputs` in `stdenv.mkDerivation`, or -- adding `pkgs.imlib2` to the `buildInputs` directly, since `pkgs` is already in-scope. - -Because `callPackage` is used to provide all necessary inputs in `nixpkgs` as well as in the `nix-build` invocation, the first approach is the one currently favored, and you should use it here: +You can add this package to your build environment by adding `imlib2` to the set of inputs to the expression in `icat.nix`, and then adding `imlib2` to the list of `buildInputs` in `stdenv.mkDerivation`: ```nix # icat.nix From 9da83aab68a06cc10ef1f5033b0247c5a07653b8 Mon Sep 17 00:00:00 2001 From: Alexander Groleau Date: Tue, 8 Aug 2023 10:50:39 -0500 Subject: [PATCH 40/74] Update source/tutorials/learning-journey/packaging-existing-software.md Co-authored-by: Silvan Mosberger --- .../tutorials/learning-journey/packaging-existing-software.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/source/tutorials/learning-journey/packaging-existing-software.md b/source/tutorials/learning-journey/packaging-existing-software.md index 657ef1abd..fe265928f 100644 --- a/source/tutorials/learning-journey/packaging-existing-software.md +++ b/source/tutorials/learning-journey/packaging-existing-software.md @@ -551,7 +551,7 @@ stdenv.mkDerivation { ``` ### Phases and Hooks -Nix package derivations are separated into [phases](https://nixos.org/manual/nixpkgs/unstable/#sec-stdenv-phases), each of which is intended to control some aspect of the build process. +Nixpkgs `stdenv.mkDerivation` derivations are separated into [phases](https://nixos.org/manual/nixpkgs/stable/#sec-stdenv-phases), each of which is intended to control some aspect of the build process. You saw earlier how the `stdenv` expected the project's `Makefile` to have an `install` target, and failed when it didn't. To fix this, you defined a custom `installPhase`, containing instructions for copying the `icat` binary to the correct output location, in effect installing it. From 3bb3ebe147ff41b1ad46c45be7fadfd3f1443b22 Mon Sep 17 00:00:00 2001 From: Alexander Groleau Date: Tue, 8 Aug 2023 10:50:53 -0500 Subject: [PATCH 41/74] Update source/tutorials/learning-journey/packaging-existing-software.md Co-authored-by: Silvan Mosberger --- .../tutorials/learning-journey/packaging-existing-software.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/source/tutorials/learning-journey/packaging-existing-software.md b/source/tutorials/learning-journey/packaging-existing-software.md index fe265928f..7411ba18a 100644 --- a/source/tutorials/learning-journey/packaging-existing-software.md +++ b/source/tutorials/learning-journey/packaging-existing-software.md @@ -553,7 +553,7 @@ stdenv.mkDerivation { ### Phases and Hooks Nixpkgs `stdenv.mkDerivation` derivations are separated into [phases](https://nixos.org/manual/nixpkgs/stable/#sec-stdenv-phases), each of which is intended to control some aspect of the build process. -You saw earlier how the `stdenv` expected the project's `Makefile` to have an `install` target, and failed when it didn't. To fix this, you defined a custom `installPhase`, containing instructions for copying the `icat` binary to the correct output location, in effect installing it. +You saw earlier how the `stdenv.mkDerivation` expected the project's `Makefile` to have an `install` target, and failed when it didn't. To fix this, you defined a custom `installPhase`, containing instructions for copying the `icat` binary to the correct output location, in effect installing it. Up to that point, the `stdenv` automatically determined the `buildPhase` information for the `icat` package. From 6bf8b0ef939faf442e62fe68bdbbfdf24a2666c2 Mon Sep 17 00:00:00 2001 From: Alexander Groleau Date: Tue, 8 Aug 2023 10:51:02 -0500 Subject: [PATCH 42/74] Update source/tutorials/learning-journey/packaging-existing-software.md Co-authored-by: Silvan Mosberger --- .../tutorials/learning-journey/packaging-existing-software.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/source/tutorials/learning-journey/packaging-existing-software.md b/source/tutorials/learning-journey/packaging-existing-software.md index 7411ba18a..344dde9f0 100644 --- a/source/tutorials/learning-journey/packaging-existing-software.md +++ b/source/tutorials/learning-journey/packaging-existing-software.md @@ -555,7 +555,7 @@ Nixpkgs `stdenv.mkDerivation` derivations are separated into [phases](https://ni You saw earlier how the `stdenv.mkDerivation` expected the project's `Makefile` to have an `install` target, and failed when it didn't. To fix this, you defined a custom `installPhase`, containing instructions for copying the `icat` binary to the correct output location, in effect installing it. -Up to that point, the `stdenv` automatically determined the `buildPhase` information for the `icat` package. +Up to that point, the `stdenv.mkDerivation` automatically determined the `buildPhase` information for the `icat` package. During derivation realisation, there are a number of shell functions ("hooks", in `nixpkgs`) which may execute in each derivation phase, which do things like set variables, source files, create directories, and so on. These are specific to each phase, and run both before and after that phase's execution, controlling the build environment and helping to prevent environment-modifying behavior defined within packages from creating sources of nondeterminism within and between Nix derivations. From 8e9a37cb0b41a6fd9008dcbd0ffb43648f03637a Mon Sep 17 00:00:00 2001 From: Alexander Groleau Date: Tue, 8 Aug 2023 11:00:19 -0500 Subject: [PATCH 43/74] Update source/tutorials/learning-journey/packaging-existing-software.md Co-authored-by: Silvan Mosberger --- .../tutorials/learning-journey/packaging-existing-software.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/source/tutorials/learning-journey/packaging-existing-software.md b/source/tutorials/learning-journey/packaging-existing-software.md index 344dde9f0..b0a891640 100644 --- a/source/tutorials/learning-journey/packaging-existing-software.md +++ b/source/tutorials/learning-journey/packaging-existing-software.md @@ -51,10 +51,11 @@ The hash cannot be known until after the tarball has been downloaded and unpacke # hello.nix { lib , stdenv +, fetchzip }: stdenv.mkDerivation { - src = builtins.fetchTarball { + src = fetchzip { url = "https://ftp.gnu.org/gnu/hello/hello-2.12.1.tar.gz"; sha256 = lib.fakeSha256; }; From 6fc7875904fd6f6aa4114390be0b854419cd6d35 Mon Sep 17 00:00:00 2001 From: Alexander Groleau Date: Tue, 8 Aug 2023 11:07:15 -0500 Subject: [PATCH 44/74] Update source/tutorials/learning-journey/packaging-existing-software.md Co-authored-by: Silvan Mosberger --- .../tutorials/learning-journey/packaging-existing-software.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/source/tutorials/learning-journey/packaging-existing-software.md b/source/tutorials/learning-journey/packaging-existing-software.md index b0a891640..c2001d9b7 100644 --- a/source/tutorials/learning-journey/packaging-existing-software.md +++ b/source/tutorials/learning-journey/packaging-existing-software.md @@ -436,7 +436,7 @@ You can see a few warnings which should be corrected in the upstream code, but t In addition to the widespread practice of prefixing a project name with `lib` to indicate the libraries of that project, in Nixpkgs it's also common to separate headers, libraries, binaries, and documentation into different output attributes of a given [derivation](https://nixos.org/manual/nix/stable/language/derivations.html). :::{note} -Determining from where to source a dependency is currently a somewhat-involved process: it helps to become familiar with searching the `nixpkgs` source for keywords, in addition to checking discussion platforms like [the official NixOS Discourse](https://discourse.nixos.org). +Determining from where to source a dependency is currently a somewhat-involved process: it helps to become familiar with searching the `nixpkgs` source for keywords. ::: You will need the `Xlib.h` headers from the `X11` C package, the Nixpkgs derivation for which is `libX11`, available in the `xorg` package set. The `Xlib` headers in turn live in the `dev` output of `xorg.libX11`. Add this to your derivation's input attribute set and to `buildInputs`: From 3e7fa59b33a01337accf89a6de7e5ccd9847530a Mon Sep 17 00:00:00 2001 From: Alexander Groleau Date: Tue, 8 Aug 2023 11:07:38 -0500 Subject: [PATCH 45/74] Update source/tutorials/learning-journey/packaging-existing-software.md Co-authored-by: Silvan Mosberger --- source/tutorials/learning-journey/packaging-existing-software.md | 1 - 1 file changed, 1 deletion(-) diff --git a/source/tutorials/learning-journey/packaging-existing-software.md b/source/tutorials/learning-journey/packaging-existing-software.md index c2001d9b7..e2e4285c8 100644 --- a/source/tutorials/learning-journey/packaging-existing-software.md +++ b/source/tutorials/learning-journey/packaging-existing-software.md @@ -433,7 +433,6 @@ error: builder for '/nix/store/bw2d4rp2k1l5rg49hds199ma2mz36x47-icat.drv' failed You can see a few warnings which should be corrected in the upstream code, but the important bit for this tutorial is `fatal error: X11/Xlib.h: No such file or directory`: another dependency is missing. -In addition to the widespread practice of prefixing a project name with `lib` to indicate the libraries of that project, in Nixpkgs it's also common to separate headers, libraries, binaries, and documentation into different output attributes of a given [derivation](https://nixos.org/manual/nix/stable/language/derivations.html). :::{note} Determining from where to source a dependency is currently a somewhat-involved process: it helps to become familiar with searching the `nixpkgs` source for keywords. From d2fe900b3960b593586fd0ebf495bc27c70bcf98 Mon Sep 17 00:00:00 2001 From: Alexander Groleau Date: Tue, 8 Aug 2023 12:22:06 -0500 Subject: [PATCH 46/74] Update source/tutorials/learning-journey/packaging-existing-software.md Co-authored-by: Silvan Mosberger --- .../tutorials/learning-journey/packaging-existing-software.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/source/tutorials/learning-journey/packaging-existing-software.md b/source/tutorials/learning-journey/packaging-existing-software.md index e2e4285c8..073d1fff7 100644 --- a/source/tutorials/learning-journey/packaging-existing-software.md +++ b/source/tutorials/learning-journey/packaging-existing-software.md @@ -459,7 +459,7 @@ stdenv.mkDerivation { sha256 = "0wyy2ksxp95vnh71ybj1bbmqd5ggp13x3mk37pzr99ljs9awy8ka"; }; - buildInputs = [ imlib2 xorg.libX11.dev ]; + buildInputs = [ imlib2 xorg.libX11 ]; } ``` From 1069b54440074ac1c5db0c129c9643f0ca35007c Mon Sep 17 00:00:00 2001 From: Alexander Groleau Date: Tue, 8 Aug 2023 12:22:17 -0500 Subject: [PATCH 47/74] Update source/tutorials/learning-journey/packaging-existing-software.md Co-authored-by: Silvan Mosberger --- .../tutorials/learning-journey/packaging-existing-software.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/source/tutorials/learning-journey/packaging-existing-software.md b/source/tutorials/learning-journey/packaging-existing-software.md index 073d1fff7..30e2e46c0 100644 --- a/source/tutorials/learning-journey/packaging-existing-software.md +++ b/source/tutorials/learning-journey/packaging-existing-software.md @@ -438,7 +438,7 @@ You can see a few warnings which should be corrected in the upstream code, but t Determining from where to source a dependency is currently a somewhat-involved process: it helps to become familiar with searching the `nixpkgs` source for keywords. ::: -You will need the `Xlib.h` headers from the `X11` C package, the Nixpkgs derivation for which is `libX11`, available in the `xorg` package set. The `Xlib` headers in turn live in the `dev` output of `xorg.libX11`. Add this to your derivation's input attribute set and to `buildInputs`: +You will need the `Xlib.h` headers from the `X11` C package, the Nixpkgs derivation for which is `libX11`, available in the `xorg` package set. Add this to your derivation's input attribute set and to `buildInputs`: ```nix # icat.nix From 3a16fb2f8d02bb28f30175d89a2b7317247129e8 Mon Sep 17 00:00:00 2001 From: Alexander Groleau Date: Tue, 8 Aug 2023 12:33:32 -0500 Subject: [PATCH 48/74] replace tabs with spaces --- .../packaging-existing-software.md | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/source/tutorials/learning-journey/packaging-existing-software.md b/source/tutorials/learning-journey/packaging-existing-software.md index 30e2e46c0..0397059f5 100644 --- a/source/tutorials/learning-journey/packaging-existing-software.md +++ b/source/tutorials/learning-journey/packaging-existing-software.md @@ -251,7 +251,7 @@ stdenv.mkDerivation { name = "icat"; src = builtins.fetchTarball { - ... + ... }; } ``` @@ -305,9 +305,9 @@ stdenv.mkDerivation { src = pkgs.fetchFromGitHub { owner = "atextor"; - repo = "icat"; - rev = "v0.5"; - sha256 = "0wyy2ksxp95vnh71ybj1bbmqd5ggp13x3mk37pzr99ljs9awy8ka"; + repo = "icat"; + rev = "v0.5"; + sha256 = "0wyy2ksxp95vnh71ybj1bbmqd5ggp13x3mk37pzr99ljs9awy8ka"; }; } ``` @@ -378,9 +378,9 @@ stdenv.mkDerivation { src = pkgs.fetchFromGitHub { owner = "atextor"; - repo = "icat"; - rev = "v0.5"; - sha256 = "0wyy2ksxp95vnh71ybj1bbmqd5ggp13x3mk37pzr99ljs9awy8ka"; + repo = "icat"; + rev = "v0.5"; + sha256 = "0wyy2ksxp95vnh71ybj1bbmqd5ggp13x3mk37pzr99ljs9awy8ka"; }; buildInputs = [ imlib2 ]; From 83abf61e3884a200db7636c65c2dc3bd1de37066 Mon Sep 17 00:00:00 2001 From: Alexander Groleau Date: Tue, 8 Aug 2023 12:34:00 -0500 Subject: [PATCH 49/74] remove unnecessary title --- .../tutorials/learning-journey/packaging-existing-software.md | 2 -- 1 file changed, 2 deletions(-) diff --git a/source/tutorials/learning-journey/packaging-existing-software.md b/source/tutorials/learning-journey/packaging-existing-software.md index 0397059f5..ecc0e069d 100644 --- a/source/tutorials/learning-journey/packaging-existing-software.md +++ b/source/tutorials/learning-journey/packaging-existing-software.md @@ -467,8 +467,6 @@ stdenv.mkDerivation { Only add the top-level `xorg` derivation to the input attrset, rather than the full `xorg.libX11.dev`, as the latter would cause a syntax error. Because Nix is lazily-evaluated, including the dependency this way is safe to do and doesn't actually include all of `xorg` into the build context. ::: - -### `buildInputs` and `nativeBuildInputs` Run the last command again: ```console $ nix-build -A icat From 55b8bfaec4cb67e6f197af21bb9a8184554f2498 Mon Sep 17 00:00:00 2001 From: Alexander Groleau Date: Tue, 8 Aug 2023 13:06:47 -0500 Subject: [PATCH 50/74] remove discussion of command line expression argument --- .../packaging-existing-software.md | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/source/tutorials/learning-journey/packaging-existing-software.md b/source/tutorials/learning-journey/packaging-existing-software.md index ecc0e069d..2ad0d0327 100644 --- a/source/tutorials/learning-journey/packaging-existing-software.md +++ b/source/tutorials/learning-journey/packaging-existing-software.md @@ -83,15 +83,9 @@ error: cannot evaluate a function that has an argument without a value ('lib') Problem: the expression in `hello.nix` is a *function*, which only produces its intended output if it is passed the correct *arguments*. ### A New Command -`lib` is available from `nixpkgs`, which must be imported with another Nix expression in order to pass it as an argument to this derivation. The `nix-build` command allows passing whole expressions as an argument following the `-E/--expr` flag, like this one: +`lib` is available from `nixpkgs`, which must be imported with another Nix expression in order to pass it as an argument to this derivation. -```console -with import {}; callPackage ./hello.nix {} -``` - -`callPackage` automatically passes attributes from `nixpkgs` to the given function (here, the one in `hello.nix`), if they match attributes required by that function's argument attrset. Here, `callPackage` will supply `lib`, and `stdenv`. - -To avoid having to execute `nix-build -E 'with import {}; callPackage ./hello.nix {}'` each time, create a `default.nix` in the same directory as `hello.nix`, with the following contents: +The recommended way to do this is to create a `default.nix` in the same directory as `hello.nix`, with the following contents: ```nix # default.nix @@ -105,6 +99,12 @@ in This allows you to use `nix-build -A hello` to realize the derivation in `hello.nix`, similar to the current convention used in `nixpkgs`. +:::{note} +`callPackage` automatically passes attributes from `nixpkgs` to the given function, if they match attributes required by that function's argument attrset. + +In this case, `callPackage` will supply `lib`, and `stdenv` to the function defined in `hello.nix`. +::: + Now run the `nix-build` command with the new argument: ```console From 50912fa0a05697f9417c69a1cd998fd501890e25 Mon Sep 17 00:00:00 2001 From: Alexander Groleau Date: Tue, 8 Aug 2023 13:20:34 -0500 Subject: [PATCH 51/74] replace fetchTarball with fetchzip --- .../packaging-existing-software.md | 23 ++++++++++++------- 1 file changed, 15 insertions(+), 8 deletions(-) diff --git a/source/tutorials/learning-journey/packaging-existing-software.md b/source/tutorials/learning-journey/packaging-existing-software.md index 2ad0d0327..b3691d5fd 100644 --- a/source/tutorials/learning-journey/packaging-existing-software.md +++ b/source/tutorials/learning-journey/packaging-existing-software.md @@ -43,9 +43,13 @@ This is a function which takes an attribute set containing `stdenv`, and produce ### Hello, World! GNU Hello is an implementation of the "hello world" program, with source code accessible [from the GNU Project's FTP server](https://ftp.gnu.org/gnu/hello/). -To begin, you will download the [latest version](https://ftp.gnu.org/gnu/hello/hello-2.12.1.tar.gz) of `hello` using `fetchTarball`, which takes the URI path to the download file and a SHA256 hash of its contents. +To begin, you will download the [latest version](https://ftp.gnu.org/gnu/hello/hello-2.12.1.tar.gz) of `hello` using `fetchzip`, which takes the URI path to the download file and a SHA256 hash of its contents. -The hash cannot be known until after the tarball has been downloaded and unpacked, but Nix will complain if the hash supplied to `fetchTarball` was incorrect, so it is common practice to supply a fake one with `lib.fakeSha256` and change the derivation definition after Nix reports the correct hash: +:::{note} +`fetchzip` can fetch [more archives](https://nixos.org/manual/nixpkgs/stable/#fetchurl) than just zip files! +::: + +The hash cannot be known until after the tarball has been downloaded and unpacked, but Nix will complain if the hash supplied to `fetchzip` was incorrect, so it is common practice to supply a fake one with `lib.fakeSha256` and change the derivation definition after Nix reports the correct hash: ```nix # hello.nix @@ -123,12 +127,13 @@ Update the file again to add a `name`: # hello.nix { lib , stdenv +, fetchzip }: stdenv.mkDerivation { name = "hello"; - src = builtins.fetchTarball { + src = fetchzip { url = "https://ftp.gnu.org/gnu/hello/hello-2.12.1.tar.gz"; sha256 = lib.fakeSha256; }; @@ -151,7 +156,7 @@ error: at /home/nix-user/hello.nix:9:3: 8| - 9| src = builtins.fetchTarball { + 9| src = fetchzip { | ^ 10| url = "https://ftp.gnu.org/gnu/hello/hello-2.12.1.tar.gz"; @@ -167,12 +172,13 @@ As expected, the incorrect file hash caused an error, and Nix helpfully provided # hello.nix { lib , stdenv +, fetchzip }: stdenv.mkDerivation { name = "hello"; - src = builtins.fetchTarball { + src = fetchzip { url = "https://ftp.gnu.org/gnu/hello/hello-2.12.1.tar.gz"; sha256 = "0xw6cr5jgi1ir13q6apvrivwmmpr5j8vbymp0x6ll0kcv6366hnn"; }; @@ -245,18 +251,19 @@ Now copy `hello.nix` to a new file, `icat.nix`, and update the `name` attribute # icat.nix { lib , stdenv +, fetchzip }: stdenv.mkDerivation { name = "icat"; - src = builtins.fetchTarball { + src = fetchzip { ... }; } ``` -Now to download the source code. `icat`'s upstream repository is hosted on [GitHub](https://github.com/atextor/icat), so you should slightly modify the previous [source fetcher](https://nixos.org/manual/nixpkgs/stable/#chap-pkgs-fetchers), this time using `pkgs.fetchFromGitHub` instead of `builtins.fetchTarball`, updating the argument attribute set to the function accordingly: +Now to download the source code. `icat`'s upstream repository is hosted on [GitHub](https://github.com/atextor/icat), so you should slightly modify the previous [source fetcher](https://nixos.org/manual/nixpkgs/stable/#chap-pkgs-fetchers), this time using `fetchFromGitHub` instead of `fetchzip`, updating the argument attribute set to the function accordingly: ```nix # icat.nix @@ -275,7 +282,7 @@ stdenv.mkDerivation { ``` ### Fetching Source from GitHub -While `fetchTarball` required `url` and `sha256` arguments, more are needed for [`fetchFromGitHub`](https://nixos.org/manual/nixpkgs/stable/#fetchfromgithub). +While `fetchzip` required `url` and `sha256` arguments, more are needed for [`fetchFromGitHub`](https://nixos.org/manual/nixpkgs/stable/#fetchfromgithub). The source is hosted on GitHub at `https://github.com/atextor/icat`, which already gives the first two arguments: - `owner`: the name of the account controlling the repository; `owner = "atextor"` From f8f420ddb5044a081e1f3fa99c061f6ff9b3f129 Mon Sep 17 00:00:00 2001 From: Alexander Groleau Date: Tue, 8 Aug 2023 13:21:27 -0500 Subject: [PATCH 52/74] remove pkgs from args --- .../packaging-existing-software.md | 30 +++++++++---------- 1 file changed, 15 insertions(+), 15 deletions(-) diff --git a/source/tutorials/learning-journey/packaging-existing-software.md b/source/tutorials/learning-journey/packaging-existing-software.md index b3691d5fd..2319841d5 100644 --- a/source/tutorials/learning-journey/packaging-existing-software.md +++ b/source/tutorials/learning-journey/packaging-existing-software.md @@ -267,15 +267,15 @@ Now to download the source code. `icat`'s upstream repository is hosted on [GitH ```nix # icat.nix -{ pkgs -, lib +{ lib , stdenv +, fetchFromGitHub }: stdenv.mkDerivation { name = "icat"; - src = pkgs.fetchFromGitHub { + src = fetchFromGitHub { ... }; } @@ -302,15 +302,15 @@ Now you can supply the correct hash to `fetchFromGitHub`: ```nix # icat.nix -{ pkgs -, lib +{ lib , stdenv +, fetchFromGitHub }: stdenv.mkDerivation { name = "icat"; - src = pkgs.fetchFromGitHub { + src = fetchFromGitHub { owner = "atextor"; repo = "icat"; rev = "v0.5"; @@ -374,16 +374,16 @@ You can add this package to your build environment by adding `imlib2` to the set ```nix # icat.nix -{ pkgs -, lib +{ lib , stdenv +, fetchFromGitHub , imlib2 }: stdenv.mkDerivation { name = "icat"; - src = pkgs.fetchFromGitHub { + src = fetchFromGitHub { owner = "atextor"; repo = "icat"; rev = "v0.5"; @@ -449,9 +449,9 @@ You will need the `Xlib.h` headers from the `X11` C package, the Nixpkgs derivat ```nix # icat.nix -{ pkgs -, lib +{ lib , stdenv +, fetchFromGitHub , imlib2 , xorg }: @@ -459,7 +459,7 @@ You will need the `Xlib.h` headers from the `X11` C package, the Nixpkgs derivat stdenv.mkDerivation { name = "icat"; - src = pkgs.fetchFromGitHub { + src = fetchFromGitHub { owner = "atextor"; repo = "icat"; rev = "v0.5"; @@ -529,9 +529,9 @@ Because the `make` step completes successfully, the `icat` executable is availab ```nix # icat.nix -{ pkgs -, lib +{ lib , stdenv +, fetchFromGitHub , imlib2 , xorg }: @@ -539,7 +539,7 @@ Because the `make` step completes successfully, the `icat` executable is availab stdenv.mkDerivation { name = "icat"; - src = pkgs.fetchFromGitHub { + src = fetchFromGitHub { owner = "atextor"; repo = "icat"; rev = "v0.5"; From a0d5460989ce318ccfac9cac57077c0d6b3be3d7 Mon Sep 17 00:00:00 2001 From: Alexander Groleau Date: Tue, 8 Aug 2023 13:21:57 -0500 Subject: [PATCH 53/74] no longer using dev output of libX11 --- .../tutorials/learning-journey/packaging-existing-software.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/source/tutorials/learning-journey/packaging-existing-software.md b/source/tutorials/learning-journey/packaging-existing-software.md index 2319841d5..916d4ffcf 100644 --- a/source/tutorials/learning-journey/packaging-existing-software.md +++ b/source/tutorials/learning-journey/packaging-existing-software.md @@ -471,7 +471,7 @@ stdenv.mkDerivation { ``` :::{note} -Only add the top-level `xorg` derivation to the input attrset, rather than the full `xorg.libX11.dev`, as the latter would cause a syntax error. Because Nix is lazily-evaluated, including the dependency this way is safe to do and doesn't actually include all of `xorg` into the build context. +Only add the top-level `xorg` derivation to the input attrset, rather than the full `xorg.libX11`, as the latter would cause a syntax error. Because Nix is lazily-evaluated, including the dependency this way is safe to do and doesn't actually include all of `xorg` into the build context. ::: Run the last command again: From 9ae846d325c81ef4ab5e6219c1725752a407d686 Mon Sep 17 00:00:00 2001 From: Alexander Groleau Date: Tue, 8 Aug 2023 13:34:11 -0500 Subject: [PATCH 54/74] remove unnecessary console output --- .../packaging-existing-software.md | 87 +------------------ 1 file changed, 3 insertions(+), 84 deletions(-) diff --git a/source/tutorials/learning-journey/packaging-existing-software.md b/source/tutorials/learning-journey/packaging-existing-software.md index 916d4ffcf..72256b060 100644 --- a/source/tutorials/learning-journey/packaging-existing-software.md +++ b/source/tutorials/learning-journey/packaging-existing-software.md @@ -144,13 +144,7 @@ and then re-run the command: ```console $ nix-build -A hello error: - … while calling the 'derivationStrict' builtin - - at /builtin/derivation.nix:9:12: (source not available) - - … while evaluating derivation 'hello' - whose name attribute is located at /nix/store/5na8c1j0cn5wls2g2d8q357cz3aqlaqj-nixos-23.05.2478.bd836ac5e5a7/nixos/pkgs/stdenv/generic/make-derivation.nix:303:7 - +... … while evaluating attribute 'src' of derivation 'hello' at /home/nix-user/hello.nix:9:3: @@ -192,18 +186,13 @@ $ nix-build -A hello this derivation will be built: /nix/store/rbq37s3r76rr77c7d8x8px7z04kw2mk7-hello.drv building '/nix/store/rbq37s3r76rr77c7d8x8px7z04kw2mk7-hello.drv'... -unpacking sources -unpacking source archive /nix/store/xdbysilxxgbs55rrdxniglqg9m1v61h4-source -source root is source -patching sources +... configuring -configure flags: --disable-dependency-tracking --prefix=/nix/store/y55w1djfnxkl2jk9w0liancp83zqb7ki-hello ... configure: creating ./config.status config.status: creating Makefile ... building -build flags: SHELL=/nix/store/7q1b1bsmxi91zci6g8714rcljl620y7f-bash-5.2-p15/bin/bash ... ``` Great news: the derivation built successfully! @@ -327,32 +316,7 @@ $ nix-build -A icat these 2 derivations will be built: /nix/store/86q9x927hsyyzfr4lcqirmsbimysi6mb-source.drv /nix/store/l5wz9inkvkf0qhl8kpl39vpg2xfm2qpy-icat.drv -these 19 paths will be fetched (4.21 MiB download, 17.65 MiB unpacked): -... -trying https://github.com/atextor/icat/archive/v0.5.tar.gz ... -unpacking source archive /build/v0.5.tar.gz -building '/nix/store/l5wz9inkvkf0qhl8kpl39vpg2xfm2qpy-icat.drv'... -unpacking sources -unpacking source archive /nix/store/rx21f6fgnmxgp1sw0wbqll9wds4xc6v0-source -source root is source -patching sources -configuring -no configure script, doing nothing -building -build flags: SHELL=/nix/store/8fv91097mbh5049i9rglc73dx6kjg3qk-bash-5.2-p15/bin/bash -gcc -c -Wall -pedantic -std=c99 -D_BSD_SOURCE -o icat.o icat.c -In file included from /nix/store/hkj250rjsvxcbr31fr1v81cv88cdfp4l-glibc-2.37-8-dev/include/bits/libc-header-start.h:33, - from /nix/store/hkj250rjsvxcbr31fr1v81cv88cdfp4l-glibc-2.37-8-dev/include/stdio.h:27, - from icat.c:31: -/nix/store/hkj250rjsvxcbr31fr1v81cv88cdfp4l-glibc-2.37-8-dev/include/features.h:195:3: warning: #warning "_BSD_SOURCE and _SVID_SOURCE are deprecated, use _DEFAULT_SOURCE" [8;;https://gcc.gnu.org/onlinedocs/gcc/Warning-Options.html#index-Wcpp-Wcpp8;;] - 195 | # warning "_BSD_SOURCE and _SVID_SOURCE are deprecated, use _DEFAULT_SOURCE" - | ^~~~~~~ -icat.c:39:10: fatal error: Imlib2.h: No such file or directory - 39 | #include - | ^~~~~~~~~~ -compilation terminated. -make: *** [Makefile:16: icat.o] Error 1 error: builder for '/nix/store/l5wz9inkvkf0qhl8kpl39vpg2xfm2qpy-icat.drv' failed with exit code 2; last 10 log lines: > from /nix/store/hkj250rjsvxcbr31fr1v81cv88cdfp4l-glibc-2.37-8-dev/include/stdio.h:27, @@ -401,28 +365,6 @@ $ nix-build -A icat this derivation will be built: /nix/store/bw2d4rp2k1l5rg49hds199ma2mz36x47-icat.drv ... -building '/nix/store/bw2d4rp2k1l5rg49hds199ma2mz36x47-icat.drv'... -unpacking sources -unpacking source archive /nix/store/rx21f6fgnmxgp1sw0wbqll9wds4xc6v0-source -source root is source -patching sources -configuring -no configure script, doing nothing -building -build flags: SHELL=/nix/store/8fv91097mbh5049i9rglc73dx6kjg3qk-bash-5.2-p15/bin/bash -gcc -c -Wall -pedantic -std=c99 -D_BSD_SOURCE -o icat.o icat.c -In file included from /nix/store/hkj250rjsvxcbr31fr1v81cv88cdfp4l-glibc-2.37-8-dev/include/bits/libc-header-start.h:33, - from /nix/store/hkj250rjsvxcbr31fr1v81cv88cdfp4l-glibc-2.37-8-dev/include/stdio.h:27, - from icat.c:31: -/nix/store/hkj250rjsvxcbr31fr1v81cv88cdfp4l-glibc-2.37-8-dev/include/features.h:195:3: warning: #warning "_BSD_SOURCE and _SVID_SOURCE are deprecated, use _DEFAULT_SOURCE" [8;;https://gcc.gnu.org/onlinedocs/gcc/Warning-Options.html#index-Wcpp-Wcpp8;;] - 195 | # warning "_BSD_SOURCE and _SVID_SOURCE are deprecated, use _DEFAULT_SOURCE" - | ^~~~~~~ -In file included from icat.c:39: -/nix/store/4fvrh0sjc8sbkbqda7dfsh7q0gxmnh9p-imlib2-1.11.1-dev/include/Imlib2.h:45:10: fatal error: X11/Xlib.h: No such file or directory - 45 | #include - | ^~~~~~~~~~~~ -compilation terminated. -make: *** [Makefile:16: icat.o] Error 1 error: builder for '/nix/store/bw2d4rp2k1l5rg49hds199ma2mz36x47-icat.drv' failed with exit code 2; last 10 log lines: > from icat.c:31: @@ -479,30 +421,7 @@ Run the last command again: $ nix-build -A icat this derivation will be built: /nix/store/x1d79ld8jxqdla5zw2b47d2sl87mf56k-icat.drv -building '/nix/store/x1d79ld8jxqdla5zw2b47d2sl87mf56k-icat.drv'... -unpacking sources -unpacking source archive /nix/store/rx21f6fgnmxgp1sw0wbqll9wds4xc6v0-source -source root is source -patching sources -configuring -no configure script, doing nothing -building -build flags: SHELL=/nix/store/8fv91097mbh5049i9rglc73dx6kjg3qk-bash-5.2-p15/bin/bash -gcc -c -Wall -pedantic -std=c99 -D_BSD_SOURCE -o icat.o icat.c -In file included from /nix/store/hkj250rjsvxcbr31fr1v81cv88cdfp4l-glibc-2.37-8-dev/include/bits/libc-header-start.h:33, - from /nix/store/hkj250rjsvxcbr31fr1v81cv88cdfp4l-glibc-2.37-8-dev/include/stdio.h:27, - from icat.c:31: -/nix/store/hkj250rjsvxcbr31fr1v81cv88cdfp4l-glibc-2.37-8-dev/include/features.h:195:3: warning: #warning "_BSD_SOURCE and _SVID_SOURCE are deprecated, use _DEFAULT_SOURCE" [8;;https://gcc.gnu.org/onlinedocs/gcc/Warning-Options.html#index-Wcpp-Wcpp8;;] - 195 | # warning "_BSD_SOURCE and _SVID_SOURCE are deprecated, use _DEFAULT_SOURCE" - | ^~~~~~~ -icat.c: In function 'main': -icat.c:319:33: warning: ignoring return value of 'write' declared with attribute 'warn_unused_result' [8;;https://gcc.gnu.org/onlinedocs/gcc/Warning-Options.html#index-Wunused-result-Wunused-result8;;] - 319 | write(tempfile, &buf, 1); - | ^~~~~~~~~~~~~~~~~~~~~~~~ -gcc -o icat icat.o -lImlib2 -installing -install flags: SHELL=/nix/store/8fv91097mbh5049i9rglc73dx6kjg3qk-bash-5.2-p15/bin/bash install -make: *** No rule to make target 'install'. Stop. +... error: builder for '/nix/store/x1d79ld8jxqdla5zw2b47d2sl87mf56k-icat.drv' failed with exit code 2; last 10 log lines: > 195 | # warning "_BSD_SOURCE and _SVID_SOURCE are deprecated, use _DEFAULT_SOURCE" From 1ccaedafd0162c7f45ba725964847890c4f386c9 Mon Sep 17 00:00:00 2001 From: Alexander Groleau Date: Tue, 8 Aug 2023 14:06:14 -0500 Subject: [PATCH 55/74] one sentence per line --- .../packaging-existing-software.md | 42 ++++++++++++++----- 1 file changed, 32 insertions(+), 10 deletions(-) diff --git a/source/tutorials/learning-journey/packaging-existing-software.md b/source/tutorials/learning-journey/packaging-existing-software.md index 72256b060..3a08ec50e 100644 --- a/source/tutorials/learning-journey/packaging-existing-software.md +++ b/source/tutorials/learning-journey/packaging-existing-software.md @@ -38,7 +38,9 @@ To start, consider this skeleton derivation: stdenv.mkDerivation { }; ``` -This is a function which takes an attribute set containing `stdenv`, and produces a derivation (which currently does nothing). As you progress through this tutorial, you will update this several times, adding more details while following the general pattern. +This is a function which takes an attribute set containing `stdenv`, and produces a derivation (which currently does nothing). + +As you progress through this tutorial, you will update this several times, adding more details while following the general pattern. ### Hello, World! GNU Hello is an implementation of the "hello world" program, with source code accessible [from the GNU Project's FTP server](https://ftp.gnu.org/gnu/hello/). @@ -277,9 +279,15 @@ The source is hosted on GitHub at `https://github.com/atextor/icat`, which alrea - `owner`: the name of the account controlling the repository; `owner = "atextor"` - `repo`: the name of the repository to fetch; `repo = "icat"` -You can navigate to the project's [Releases page](https://github.com/atextor/icat/releases) to find a suitable `rev`, such as the git commit hash or tag (e.g. `v1.0`) corresponding to the release you want to fetch. In this case, the latest release tag is `v0.5`. +You can navigate to the project's [Releases page](https://github.com/atextor/icat/releases) to find a suitable `rev`, such as the git commit hash or tag (e.g. `v1.0`) corresponding to the release you want to fetch. + +In this case, the latest release tag is `v0.5`. + +As in the `hello` example, a hash must also be supplied. -As in the `hello` example, a hash must also be supplied. This time, instead of using `lib.fakeSha256` and letting `nix-build` report the correct one in an error, you can fetch the correct hash in the first place with the `nix-prefetch-url` command. You need the SHA256 hash of the *contents* of the tarball (as opposed to the hash of the tarball file itself), so you will need to pass the `--unpack` and `--type sha256` arguments too: +This time, instead of using `lib.fakeSha256` and letting `nix-build` report the correct one in an error, you can fetch the correct hash in the first place with the `nix-prefetch-url` command. + +You need the SHA256 hash of the *contents* of the tarball (as opposed to the hash of the tarball file itself), so you will need to pass the `--unpack` and `--type sha256` arguments too: ```console $ nix-prefetch-url --unpack https://github.com/atextor/icat/archive/refs/tags/v0.5.tar.gz --type sha256 @@ -332,7 +340,9 @@ error: builder for '/nix/store/l5wz9inkvkf0qhl8kpl39vpg2xfm2qpy-icat.drv' failed For full logs, run 'nix log /nix/store/l5wz9inkvkf0qhl8kpl39vpg2xfm2qpy-icat.drv'. ``` -A compiler error! The `icat` source was pulled from GitHub, and Nix tried to build what it found, but compilation failed due to a missing dependency: the `imlib2` header. If you [search for `imlib2` on search.nixos.org](https://search.nixos.org/packages?channel=23.05&from=0&size=50&sort=relevance&type=packages&query=imlib2), you'll find that `imlib2` is already in `nixpkgs`. +A compiler error! The `icat` source was pulled from GitHub, and Nix tried to build what it found, but compilation failed due to a missing dependency: the `imlib2` header. + +If you [search for `imlib2` on search.nixos.org](https://search.nixos.org/packages?channel=23.05&from=0&size=50&sort=relevance&type=packages&query=imlib2), you'll find that `imlib2` is already in `nixpkgs`. You can add this package to your build environment by adding `imlib2` to the set of inputs to the expression in `icat.nix`, and then adding `imlib2` to the list of `buildInputs` in `stdenv.mkDerivation`: @@ -387,7 +397,9 @@ You can see a few warnings which should be corrected in the upstream code, but t Determining from where to source a dependency is currently a somewhat-involved process: it helps to become familiar with searching the `nixpkgs` source for keywords. ::: -You will need the `Xlib.h` headers from the `X11` C package, the Nixpkgs derivation for which is `libX11`, available in the `xorg` package set. Add this to your derivation's input attribute set and to `buildInputs`: +You will need the `Xlib.h` headers from the `X11` C package, the Nixpkgs derivation for which is `libX11`, available in the `xorg` package set. + +Add this to your derivation's input attribute set and to `buildInputs`: ```nix # icat.nix @@ -413,7 +425,9 @@ stdenv.mkDerivation { ``` :::{note} -Only add the top-level `xorg` derivation to the input attrset, rather than the full `xorg.libX11`, as the latter would cause a syntax error. Because Nix is lazily-evaluated, including the dependency this way is safe to do and doesn't actually include all of `xorg` into the build context. +Only add the top-level `xorg` derivation to the input attrset, rather than the full `xorg.libX11`, as the latter would cause a syntax error. + +Because Nix is lazily-evaluated, including the dependency this way is safe to do and doesn't actually include all of `xorg` into the build context. ::: Run the last command again: @@ -440,11 +454,15 @@ error: builder for '/nix/store/x1d79ld8jxqdla5zw2b47d2sl87mf56k-icat.drv' failed The missing dependency error is solved, but there is now another problem: `make: *** No rule to make target 'install'. Stop.` ### installPhase -The `stdenv` is automatically working with the `Makefile` that comes with `icat`: you can see in the console output that `configure` and `make` are executed without issue, so the `icat` binary is compiling successfully. The failure occurs when the `stdenv` attempts to run `make install`: the `Makefile` included in the project happens to lack an `install` target, and the `README` in the `icat` repository only mentions using `make` to build the tool, leaving the installation step up to users. +The `stdenv` is automatically working with the `Makefile` that comes with `icat`: you can see in the console output that `configure` and `make` are executed without issue, so the `icat` binary is compiling successfully. + +The failure occurs when the `stdenv` attempts to run `make install`: the `Makefile` included in the project happens to lack an `install` target, and the `README` in the `icat` repository only mentions using `make` to build the tool, leaving the installation step up to users. To add this step to your derivation, use the [`installPhase` attribute](https://nixos.org/manual/nixpkgs/stable/#ssec-install-phase), which contains a list of command strings to execute to perform the installation. -Because the `make` step completes successfully, the `icat` executable is available in the build directory, and you only need to copy it from there to the output directory. In Nix, this location is stored in the `$out` variable, accessible in the derivation's component scripts; create a `bin` directory within that and copy the `icat` binary there: +Because the `make` step completes successfully, the `icat` executable is available in the build directory, and you only need to copy it from there to the output directory. + +In Nix, this location is stored in the `$out` variable, accessible in the derivation's component scripts; create a `bin` directory within that and copy the `icat` binary there: ```nix # icat.nix @@ -477,11 +495,15 @@ stdenv.mkDerivation { ### Phases and Hooks Nixpkgs `stdenv.mkDerivation` derivations are separated into [phases](https://nixos.org/manual/nixpkgs/stable/#sec-stdenv-phases), each of which is intended to control some aspect of the build process. -You saw earlier how the `stdenv.mkDerivation` expected the project's `Makefile` to have an `install` target, and failed when it didn't. To fix this, you defined a custom `installPhase`, containing instructions for copying the `icat` binary to the correct output location, in effect installing it. +You saw earlier how the `stdenv.mkDerivation` expected the project's `Makefile` to have an `install` target, and failed when it didn't. + +To fix this, you defined a custom `installPhase`, containing instructions for copying the `icat` binary to the correct output location, in effect installing it. Up to that point, the `stdenv.mkDerivation` automatically determined the `buildPhase` information for the `icat` package. -During derivation realisation, there are a number of shell functions ("hooks", in `nixpkgs`) which may execute in each derivation phase, which do things like set variables, source files, create directories, and so on. These are specific to each phase, and run both before and after that phase's execution, controlling the build environment and helping to prevent environment-modifying behavior defined within packages from creating sources of nondeterminism within and between Nix derivations. +During derivation realisation, there are a number of shell functions ("hooks", in `nixpkgs`) which may execute in each derivation phase, which do things like set variables, source files, create directories, and so on. + +These are specific to each phase, and run both before and after that phase's execution, controlling the build environment and helping to prevent environment-modifying behavior defined within packages from creating sources of nondeterminism within and between Nix derivations. It's good practice when packaging software with Nix to include calls to these hooks in the derivation phases you define, even when you don't make direct use of them; this facilitates easy [overriding](https://nixos.org/manual/nixpkgs/stable/#chap-overrides) of specific parts of the derivation later, in addition to the previously-mentioned reproducibility benefits. From 7ede9c511520b2dab36f7c8c23d6b430712826b8 Mon Sep 17 00:00:00 2001 From: Alexander Groleau Date: Wed, 9 Aug 2023 09:16:04 -0500 Subject: [PATCH 56/74] Update source/tutorials/learning-journey/packaging-existing-software.md Co-authored-by: Zach Mitchell, PhD --- .../tutorials/learning-journey/packaging-existing-software.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/source/tutorials/learning-journey/packaging-existing-software.md b/source/tutorials/learning-journey/packaging-existing-software.md index 3a08ec50e..a64b74ad2 100644 --- a/source/tutorials/learning-journey/packaging-existing-software.md +++ b/source/tutorials/learning-journey/packaging-existing-software.md @@ -13,7 +13,7 @@ myst: One of Nix's primary use-cases is in addressing common difficulties encountered while packaging software, like *managing dependencies*. -In the long term, Nix helps tremendously in alleviating that stress, but when *first* (re)packaging existing software with Nix, it's common to encounter missing dependencies preventing builds from succeeding. +In the long term, Nix helps tremendously in alleviating that stress, but when *first* packaging existing software with Nix, it's common to encounter missing dependencies preventing builds from succeeding. In this tutorial, you'll create your first Nix derivations to package C/C++ software, taking advantage of the [`nixpkgs` `stdenv`](https://nixos.org/manual/nixpkgs/stable/#chap-stdenv) which automates much of the work of building self-contained C/C++ packages. From 24fba428cbf996bede2a0b1830e40bf4457ea420 Mon Sep 17 00:00:00 2001 From: Alexander Groleau Date: Wed, 9 Aug 2023 09:53:59 -0500 Subject: [PATCH 57/74] Update source/tutorials/learning-journey/packaging-existing-software.md Co-authored-by: Zach Mitchell, PhD --- .../tutorials/learning-journey/packaging-existing-software.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/source/tutorials/learning-journey/packaging-existing-software.md b/source/tutorials/learning-journey/packaging-existing-software.md index a64b74ad2..dd47010a6 100644 --- a/source/tutorials/learning-journey/packaging-existing-software.md +++ b/source/tutorials/learning-journey/packaging-existing-software.md @@ -11,7 +11,7 @@ myst: # Packaging Existing Software With Nix -One of Nix's primary use-cases is in addressing common difficulties encountered while packaging software, like *managing dependencies*. +One of Nix's primary use-cases is in addressing common difficulties encountered while packaging software, like managing dependencies. In the long term, Nix helps tremendously in alleviating that stress, but when *first* packaging existing software with Nix, it's common to encounter missing dependencies preventing builds from succeeding. From a7af1804866839812562d463f64629ad86a76566 Mon Sep 17 00:00:00 2001 From: Alexander Groleau Date: Wed, 9 Aug 2023 09:54:27 -0500 Subject: [PATCH 58/74] Update source/tutorials/learning-journey/packaging-existing-software.md Co-authored-by: Zach Mitchell, PhD --- .../tutorials/learning-journey/packaging-existing-software.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/source/tutorials/learning-journey/packaging-existing-software.md b/source/tutorials/learning-journey/packaging-existing-software.md index dd47010a6..aa226be79 100644 --- a/source/tutorials/learning-journey/packaging-existing-software.md +++ b/source/tutorials/learning-journey/packaging-existing-software.md @@ -427,7 +427,7 @@ stdenv.mkDerivation { :::{note} Only add the top-level `xorg` derivation to the input attrset, rather than the full `xorg.libX11`, as the latter would cause a syntax error. -Because Nix is lazily-evaluated, including the dependency this way is safe to do and doesn't actually include all of `xorg` into the build context. +Because Nix is lazily-evaluated, using `xorg.libX11` means that we only include the `libX11` attribute and the derivation doesn't actually include all of `xorg` into the build context. ::: Run the last command again: From 8998be56e4da9b1309f20ca0be4e1f9ea85c03ca Mon Sep 17 00:00:00 2001 From: Alexander Groleau Date: Wed, 9 Aug 2023 09:55:09 -0500 Subject: [PATCH 59/74] Update source/tutorials/learning-journey/packaging-existing-software.md Co-authored-by: Zach Mitchell, PhD --- .../tutorials/learning-journey/packaging-existing-software.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/source/tutorials/learning-journey/packaging-existing-software.md b/source/tutorials/learning-journey/packaging-existing-software.md index aa226be79..c1aab7664 100644 --- a/source/tutorials/learning-journey/packaging-existing-software.md +++ b/source/tutorials/learning-journey/packaging-existing-software.md @@ -462,7 +462,8 @@ To add this step to your derivation, use the [`installPhase` attribute](https:// Because the `make` step completes successfully, the `icat` executable is available in the build directory, and you only need to copy it from there to the output directory. -In Nix, this location is stored in the `$out` variable, accessible in the derivation's component scripts; create a `bin` directory within that and copy the `icat` binary there: +In Nix, the output directory is stored in the `$out` variable, accessible in the derivation's component scripts. +Create a `bin` directory within the `$out` directory and copy the `icat` binary there: ```nix # icat.nix From 4ed39f5efff08d414b18aef874da3a15c2587687 Mon Sep 17 00:00:00 2001 From: Alexander Groleau Date: Wed, 9 Aug 2023 09:55:37 -0500 Subject: [PATCH 60/74] Update source/tutorials/learning-journey/packaging-existing-software.md Co-authored-by: Zach Mitchell, PhD --- .../tutorials/learning-journey/packaging-existing-software.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/source/tutorials/learning-journey/packaging-existing-software.md b/source/tutorials/learning-journey/packaging-existing-software.md index c1aab7664..35d884bbc 100644 --- a/source/tutorials/learning-journey/packaging-existing-software.md +++ b/source/tutorials/learning-journey/packaging-existing-software.md @@ -17,7 +17,7 @@ In the long term, Nix helps tremendously in alleviating that stress, but when *f In this tutorial, you'll create your first Nix derivations to package C/C++ software, taking advantage of the [`nixpkgs` `stdenv`](https://nixos.org/manual/nixpkgs/stable/#chap-stdenv) which automates much of the work of building self-contained C/C++ packages. -The tutorial begins by considering `hello`, an implementation of "hello world" which only requires dependencies already in `stdenv`. +The tutorial begins by considering `hello`, an implementation of "hello world" which only requires dependencies provided by `stdenv`. Next, you will build more complex packages with their own dependencies, leading you to use additional derivation features. From 37f5fd30307630fd3395c18968deda99064ab8f0 Mon Sep 17 00:00:00 2001 From: Alexander Groleau Date: Wed, 9 Aug 2023 09:56:31 -0500 Subject: [PATCH 61/74] Update source/tutorials/learning-journey/packaging-existing-software.md Co-authored-by: Zach Mitchell, PhD --- .../tutorials/learning-journey/packaging-existing-software.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/source/tutorials/learning-journey/packaging-existing-software.md b/source/tutorials/learning-journey/packaging-existing-software.md index 35d884bbc..1cfb6c836 100644 --- a/source/tutorials/learning-journey/packaging-existing-software.md +++ b/source/tutorials/learning-journey/packaging-existing-software.md @@ -15,7 +15,7 @@ One of Nix's primary use-cases is in addressing common difficulties encountered In the long term, Nix helps tremendously in alleviating that stress, but when *first* packaging existing software with Nix, it's common to encounter missing dependencies preventing builds from succeeding. -In this tutorial, you'll create your first Nix derivations to package C/C++ software, taking advantage of the [`nixpkgs` `stdenv`](https://nixos.org/manual/nixpkgs/stable/#chap-stdenv) which automates much of the work of building self-contained C/C++ packages. +In this tutorial, you'll create your first Nix derivations to package C/C++ software, taking advantage of the [`nixpkgs stdenv`](https://nixos.org/manual/nixpkgs/stable/#chap-stdenv) which automates much of the work of building self-contained C/C++ packages. The tutorial begins by considering `hello`, an implementation of "hello world" which only requires dependencies provided by `stdenv`. From 2ace3c6e1c00f0ce032546beeba8558473557e79 Mon Sep 17 00:00:00 2001 From: Alexander Groleau Date: Wed, 9 Aug 2023 10:40:04 -0500 Subject: [PATCH 62/74] Update source/tutorials/learning-journey/packaging-existing-software.md Co-authored-by: Zach Mitchell, PhD --- .../tutorials/learning-journey/packaging-existing-software.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/source/tutorials/learning-journey/packaging-existing-software.md b/source/tutorials/learning-journey/packaging-existing-software.md index 1cfb6c836..2575850d5 100644 --- a/source/tutorials/learning-journey/packaging-existing-software.md +++ b/source/tutorials/learning-journey/packaging-existing-software.md @@ -199,7 +199,8 @@ building ``` Great news: the derivation built successfully! -The console output shows that `configure` was called, which produced a `Makefile` that was then used to build the project; it wasn't necessary to write any build instructions in this case, because the `stdenv` build system is based on `autoconf`, which automatically detected the structure of the project directory. +The console output shows that `configure` was called, which produced a `Makefile` that was then used to build the project. +It wasn't necessary to write any build instructions in this case because the `stdenv` build system is based on `autoconf`, which automatically detected the structure of the project directory. ### Build Result Check your working directory for the result: From b374799ca22edb9d79c0cd7b8a1f9f03856efc17 Mon Sep 17 00:00:00 2001 From: Alexander Groleau Date: Wed, 9 Aug 2023 10:42:38 -0500 Subject: [PATCH 63/74] Update source/tutorials/learning-journey/packaging-existing-software.md Co-authored-by: Zach Mitchell, PhD --- .../tutorials/learning-journey/packaging-existing-software.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/source/tutorials/learning-journey/packaging-existing-software.md b/source/tutorials/learning-journey/packaging-existing-software.md index 2575850d5..a472f841d 100644 --- a/source/tutorials/learning-journey/packaging-existing-software.md +++ b/source/tutorials/learning-journey/packaging-existing-software.md @@ -210,7 +210,7 @@ $ ls default.nix hello.nix result ``` -This result is a symbolic link to a Nix store location containing the built binary; you can call `./result/bin/hello` to execute this program: +This `result` is a symbolic link to a Nix store location containing the built binary; you can call `./result/bin/hello` to execute this program: ```console $ ./result/bin/hello From 42cf37cd43cb3fb9f141fb7a8b619e98b322a79b Mon Sep 17 00:00:00 2001 From: Alexander Groleau Date: Wed, 9 Aug 2023 10:43:16 -0500 Subject: [PATCH 64/74] Update source/tutorials/learning-journey/packaging-existing-software.md Co-authored-by: Zach Mitchell, PhD --- .../tutorials/learning-journey/packaging-existing-software.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/source/tutorials/learning-journey/packaging-existing-software.md b/source/tutorials/learning-journey/packaging-existing-software.md index a472f841d..587eee08f 100644 --- a/source/tutorials/learning-journey/packaging-existing-software.md +++ b/source/tutorials/learning-journey/packaging-existing-software.md @@ -255,7 +255,8 @@ stdenv.mkDerivation { } ``` -Now to download the source code. `icat`'s upstream repository is hosted on [GitHub](https://github.com/atextor/icat), so you should slightly modify the previous [source fetcher](https://nixos.org/manual/nixpkgs/stable/#chap-pkgs-fetchers), this time using `fetchFromGitHub` instead of `fetchzip`, updating the argument attribute set to the function accordingly: +Now to download the source code. +`icat`'s upstream repository is hosted on [GitHub](https://github.com/atextor/icat), so you should modify the previous [source fetcher](https://nixos.org/manual/nixpkgs/stable/#chap-pkgs-fetchers), this time using `fetchFromGitHub` instead of `fetchzip`, updating the argument attribute set to the function accordingly: ```nix # icat.nix From 2ab3c4b4ad148e03f983c54da877534e5e8cba18 Mon Sep 17 00:00:00 2001 From: Alexander Groleau Date: Wed, 9 Aug 2023 10:44:38 -0500 Subject: [PATCH 65/74] Update source/tutorials/learning-journey/packaging-existing-software.md Co-authored-by: Zach Mitchell, PhD --- .../tutorials/learning-journey/packaging-existing-software.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/source/tutorials/learning-journey/packaging-existing-software.md b/source/tutorials/learning-journey/packaging-existing-software.md index 587eee08f..48df6882f 100644 --- a/source/tutorials/learning-journey/packaging-existing-software.md +++ b/source/tutorials/learning-journey/packaging-existing-software.md @@ -342,7 +342,8 @@ error: builder for '/nix/store/l5wz9inkvkf0qhl8kpl39vpg2xfm2qpy-icat.drv' failed For full logs, run 'nix log /nix/store/l5wz9inkvkf0qhl8kpl39vpg2xfm2qpy-icat.drv'. ``` -A compiler error! The `icat` source was pulled from GitHub, and Nix tried to build what it found, but compilation failed due to a missing dependency: the `imlib2` header. +A compiler error! +The `icat` source was pulled from GitHub, and Nix tried to build what it found, but compilation failed due to a missing dependency: the `imlib2` header. If you [search for `imlib2` on search.nixos.org](https://search.nixos.org/packages?channel=23.05&from=0&size=50&sort=relevance&type=packages&query=imlib2), you'll find that `imlib2` is already in `nixpkgs`. From c714c04a60b939221c4433a0f7d871a0069673e5 Mon Sep 17 00:00:00 2001 From: Alexander Groleau Date: Wed, 9 Aug 2023 10:51:42 -0500 Subject: [PATCH 66/74] Update source/tutorials/learning-journey/packaging-existing-software.md Co-authored-by: Zach Mitchell, PhD --- .../tutorials/learning-journey/packaging-existing-software.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/source/tutorials/learning-journey/packaging-existing-software.md b/source/tutorials/learning-journey/packaging-existing-software.md index 48df6882f..e72bc119e 100644 --- a/source/tutorials/learning-journey/packaging-existing-software.md +++ b/source/tutorials/learning-journey/packaging-existing-software.md @@ -371,7 +371,7 @@ stdenv.mkDerivation { } ``` -Another error, but compilation proceeds further this time: +Run `nix-build -A icat` again and you'll encounter another error, but compilation proceeds further this time: ```console $ nix-build -A icat From 7a9199115fa6612e0927381566e63663a69e98e1 Mon Sep 17 00:00:00 2001 From: Alexander Groleau Date: Wed, 9 Aug 2023 10:54:41 -0500 Subject: [PATCH 67/74] Update source/tutorials/learning-journey/packaging-existing-software.md Co-authored-by: Zach Mitchell, PhD --- .../tutorials/learning-journey/packaging-existing-software.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/source/tutorials/learning-journey/packaging-existing-software.md b/source/tutorials/learning-journey/packaging-existing-software.md index e72bc119e..57af13ec1 100644 --- a/source/tutorials/learning-journey/packaging-existing-software.md +++ b/source/tutorials/learning-journey/packaging-existing-software.md @@ -501,7 +501,7 @@ Nixpkgs `stdenv.mkDerivation` derivations are separated into [phases](https://ni You saw earlier how the `stdenv.mkDerivation` expected the project's `Makefile` to have an `install` target, and failed when it didn't. -To fix this, you defined a custom `installPhase`, containing instructions for copying the `icat` binary to the correct output location, in effect installing it. +To fix this, you defined a custom `installPhase` containing instructions for copying the `icat` binary to the correct output location, in effect installing it. Up to that point, the `stdenv.mkDerivation` automatically determined the `buildPhase` information for the `icat` package. From 582553cf3c778792ce0a8c721243b4b5700207c6 Mon Sep 17 00:00:00 2001 From: Alexander Groleau Date: Wed, 9 Aug 2023 10:56:50 -0500 Subject: [PATCH 68/74] Update source/tutorials/learning-journey/packaging-existing-software.md Co-authored-by: Zach Mitchell, PhD --- .../tutorials/learning-journey/packaging-existing-software.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/source/tutorials/learning-journey/packaging-existing-software.md b/source/tutorials/learning-journey/packaging-existing-software.md index 57af13ec1..1dc940ccd 100644 --- a/source/tutorials/learning-journey/packaging-existing-software.md +++ b/source/tutorials/learning-journey/packaging-existing-software.md @@ -524,7 +524,7 @@ You should now adjust your `installPhase` to call the appropriate hooks: ''; ... ``` - +### A successful build Running the `nix-build` command once more will finally do what you want, and more safely than before; you can `ls` in the local directory to find a `result` symlink to a location in the Nix store: ```console From 06285c2e72e90cc81453dd427c241ee21f374242 Mon Sep 17 00:00:00 2001 From: Alexander Groleau Date: Thu, 10 Aug 2023 08:07:28 -0500 Subject: [PATCH 69/74] Update source/tutorials/learning-journey/packaging-existing-software.md Co-authored-by: Silvan Mosberger --- .../tutorials/learning-journey/packaging-existing-software.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/source/tutorials/learning-journey/packaging-existing-software.md b/source/tutorials/learning-journey/packaging-existing-software.md index 1dc940ccd..fd4c7c43e 100644 --- a/source/tutorials/learning-journey/packaging-existing-software.md +++ b/source/tutorials/learning-journey/packaging-existing-software.md @@ -24,7 +24,8 @@ Next, you will build more complex packages with their own dependencies, leading You'll encounter and address Nix error messages, build failures, and a host of other issues, developing your iterative debugging techniques along the way. :::{note} -An important point of clarification: the term "package" is used conventionally by analogy to other systems, although the term does not refer to a proper concept in Nix. +A _package_ is an informally defined Nixpkgs concept referring to a Nix derivation representing an installation of some project. +Packages have mostly standardised attributes and output layouts, allowing them to be discovered in searches and installed into environments alongside other packages. For the purposes of this tutorial, "package" means something like "result of a derivation"; this is the artifact you or others will use, as a consequence of having "packaged existing software with Nix". ::: From 8e38972d40a81f7cb37150a1398fbb2efb934be0 Mon Sep 17 00:00:00 2001 From: Alexander Groleau Date: Thu, 10 Aug 2023 08:56:24 -0500 Subject: [PATCH 70/74] Update source/tutorials/learning-journey/packaging-existing-software.md Co-authored-by: Zach Mitchell, PhD --- .../tutorials/learning-journey/packaging-existing-software.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/source/tutorials/learning-journey/packaging-existing-software.md b/source/tutorials/learning-journey/packaging-existing-software.md index fd4c7c43e..3c42791ab 100644 --- a/source/tutorials/learning-journey/packaging-existing-software.md +++ b/source/tutorials/learning-journey/packaging-existing-software.md @@ -279,7 +279,7 @@ stdenv.mkDerivation { While `fetchzip` required `url` and `sha256` arguments, more are needed for [`fetchFromGitHub`](https://nixos.org/manual/nixpkgs/stable/#fetchfromgithub). The source is hosted on GitHub at `https://github.com/atextor/icat`, which already gives the first two arguments: -- `owner`: the name of the account controlling the repository; `owner = "atextor"` +- `owner`: the name of the account controlling the repository; `owner = "atextor";` - `repo`: the name of the repository to fetch; `repo = "icat"` You can navigate to the project's [Releases page](https://github.com/atextor/icat/releases) to find a suitable `rev`, such as the git commit hash or tag (e.g. `v1.0`) corresponding to the release you want to fetch. From b9056f6d3af0742e117dafec2ed1cb6fafdb1255 Mon Sep 17 00:00:00 2001 From: Alexander Groleau Date: Thu, 10 Aug 2023 08:56:44 -0500 Subject: [PATCH 71/74] Update source/tutorials/learning-journey/packaging-existing-software.md Co-authored-by: Zach Mitchell, PhD --- .../tutorials/learning-journey/packaging-existing-software.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/source/tutorials/learning-journey/packaging-existing-software.md b/source/tutorials/learning-journey/packaging-existing-software.md index 3c42791ab..85f003797 100644 --- a/source/tutorials/learning-journey/packaging-existing-software.md +++ b/source/tutorials/learning-journey/packaging-existing-software.md @@ -280,7 +280,7 @@ While `fetchzip` required `url` and `sha256` arguments, more are needed for [`fe The source is hosted on GitHub at `https://github.com/atextor/icat`, which already gives the first two arguments: - `owner`: the name of the account controlling the repository; `owner = "atextor";` -- `repo`: the name of the repository to fetch; `repo = "icat"` +- `repo`: the name of the repository to fetch; `repo = "icat";` You can navigate to the project's [Releases page](https://github.com/atextor/icat/releases) to find a suitable `rev`, such as the git commit hash or tag (e.g. `v1.0`) corresponding to the release you want to fetch. From f1fb5053984a72829c0349cd0c6b9300a8cb3595 Mon Sep 17 00:00:00 2001 From: Alexander Groleau Date: Thu, 10 Aug 2023 08:58:19 -0500 Subject: [PATCH 72/74] Update source/tutorials/learning-journey/packaging-existing-software.md Co-authored-by: Zach Mitchell, PhD --- .../tutorials/learning-journey/packaging-existing-software.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/source/tutorials/learning-journey/packaging-existing-software.md b/source/tutorials/learning-journey/packaging-existing-software.md index 85f003797..a590cb6de 100644 --- a/source/tutorials/learning-journey/packaging-existing-software.md +++ b/source/tutorials/learning-journey/packaging-existing-software.md @@ -500,7 +500,7 @@ stdenv.mkDerivation { ### Phases and Hooks Nixpkgs `stdenv.mkDerivation` derivations are separated into [phases](https://nixos.org/manual/nixpkgs/stable/#sec-stdenv-phases), each of which is intended to control some aspect of the build process. -You saw earlier how the `stdenv.mkDerivation` expected the project's `Makefile` to have an `install` target, and failed when it didn't. +You saw earlier how `stdenv.mkDerivation` expected the project's `Makefile` to have an `install` target, and failed when it didn't. To fix this, you defined a custom `installPhase` containing instructions for copying the `icat` binary to the correct output location, in effect installing it. From 7c950a309ef55388f771ada4ec675f8a4e55ba2c Mon Sep 17 00:00:00 2001 From: Alexander Groleau Date: Thu, 10 Aug 2023 10:21:27 -0500 Subject: [PATCH 73/74] remove `name` error encounter --- .../packaging-existing-software.md | 43 ++++++------------- 1 file changed, 12 insertions(+), 31 deletions(-) diff --git a/source/tutorials/learning-journey/packaging-existing-software.md b/source/tutorials/learning-journey/packaging-existing-software.md index a590cb6de..b6ff97b8b 100644 --- a/source/tutorials/learning-journey/packaging-existing-software.md +++ b/source/tutorials/learning-journey/packaging-existing-software.md @@ -46,7 +46,16 @@ As you progress through this tutorial, you will update this several times, addin ### Hello, World! GNU Hello is an implementation of the "hello world" program, with source code accessible [from the GNU Project's FTP server](https://ftp.gnu.org/gnu/hello/). -To begin, you will download the [latest version](https://ftp.gnu.org/gnu/hello/hello-2.12.1.tar.gz) of `hello` using `fetchzip`, which takes the URI path to the download file and a SHA256 hash of its contents. +To begin, you should add a `name` attribute to the set passed to `mkDerivation`; every derivation needs a name, and Nix will throw `error: derivation name missing` without one. + +```diff +... +stdenv.mkDerivation { ++ name = "hello"; +... +``` + +Next, you will download the [latest version](https://ftp.gnu.org/gnu/hello/hello-2.12.1.tar.gz) of `hello` using `fetchzip`, which takes the URI path to the download file and a SHA256 hash of its contents. :::{note} `fetchzip` can fetch [more archives](https://nixos.org/manual/nixpkgs/stable/#fetchurl) than just zip files! @@ -62,6 +71,8 @@ The hash cannot be known until after the tarball has been downloaded and unpacke }: stdenv.mkDerivation { + name = "hello"; + src = fetchzip { url = "https://ftp.gnu.org/gnu/hello/hello-2.12.1.tar.gz"; sha256 = lib.fakeSha256; @@ -114,36 +125,6 @@ In this case, `callPackage` will supply `lib`, and `stdenv` to the function defi Now run the `nix-build` command with the new argument: -```console -$ nix-build -A hello -error: derivation name missing -``` - -This new failure occurs with the *derivation*, further down in the file than the initial error on line 2 about the `lib` argument not having a value; the previous error was successfully resolved by changing the expression passed to `nix-build`. - -### Naming a Derivation -Every derivation needs a `name` attribute, which must either be set directly or constructed by `mkDerivation` from `pname` and `version` attributes, if they exist. - -Update the file again to add a `name`: - -```nix -# hello.nix -{ lib -, stdenv -, fetchzip -}: - -stdenv.mkDerivation { - name = "hello"; - - src = fetchzip { - url = "https://ftp.gnu.org/gnu/hello/hello-2.12.1.tar.gz"; - sha256 = lib.fakeSha256; - }; -} -``` -and then re-run the command: - ```console $ nix-build -A hello error: From 6bd4567302964854c7bb3fcda0a65bd96ec8ffbb Mon Sep 17 00:00:00 2001 From: Alexander Groleau Date: Thu, 10 Aug 2023 10:29:41 -0500 Subject: [PATCH 74/74] recommend nix-locate/nix-index --- .../tutorials/learning-journey/packaging-existing-software.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/source/tutorials/learning-journey/packaging-existing-software.md b/source/tutorials/learning-journey/packaging-existing-software.md index b6ff97b8b..3f8c15f6c 100644 --- a/source/tutorials/learning-journey/packaging-existing-software.md +++ b/source/tutorials/learning-journey/packaging-existing-software.md @@ -380,6 +380,8 @@ You can see a few warnings which should be corrected in the upstream code, but t :::{note} Determining from where to source a dependency is currently a somewhat-involved process: it helps to become familiar with searching the `nixpkgs` source for keywords. + +Consider using `nix-locate` from the [`nix-index`](https://github.com/nix-community/nix-index) tool to find derivations that provide what you need. ::: You will need the `Xlib.h` headers from the `X11` C package, the Nixpkgs derivation for which is `libX11`, available in the `xorg` package set.