diff --git a/README.md b/README.md index 3cc19e2af4..65ab65cf8c 100644 --- a/README.md +++ b/README.md @@ -48,4 +48,3 @@ The intention is to integrate many existing 2nix converters into this framework, ### Community [πŸ‘‰ matrix chat room](https://matrix.to/#/#dream2nix:nixos.org) - diff --git a/docs/src/SUMMARY.md b/docs/src/SUMMARY.md index 7ba2605d9a..ec9ead80cc 100644 --- a/docs/src/SUMMARY.md +++ b/docs/src/SUMMARY.md @@ -1,40 +1,8 @@ - [Introduction](./intro.md) - -# Guides -- [Python: getting started in 10 minutes](./guides/getting-started-python.md) -- [Node.js: quickstart](./guides/getting-started-nodejs.md) - -# Subsystems -- [Rust](./subsystems/rust.md) -- [Python](./subsystems/python.md) -- [Node.js](./subsystems/node.md) -- [Haskell](./subsystems/haskell.md) -- [PHP](./subsystems/php.md) - -# Concepts / API -- [Architecture](./intro/architecture.md) -- [Translators](./intro/translators.md) -- [Generic Lockfile](./api/generic-lock.md) -- [Indexers](./intro/indexers.md) -- [Fetchers](./intro/fetchers.md) -- [Builders]() -- [Override system](./intro/override-system.md) - -# Correlation with nixpkgs -- [Nixpkgs improvements](./intro/nixpkgs-improvements.md) - -# Contributing -- [Extending dream2nix](./extending-dream2nix.md) -- [Contributing](./contributing/contributing.md) - -# Development Roundups -- [April - June 2022](./development-roundups/2022-april-june.md) -- [July - September 2022](./development-roundups/2022-july-september.md) - -# WIP -- [v1 API](./v1-api/summary.md) - - [problems of the current dream2nix](./v1-api/problems.md) +# dream2nix concept +- [draft for modules based API](./v1-api/summary.md) + - [problems of the legacy dream2nix](./v1-api/problems.md) - [users of dream2nix](./v1-api/users.md) - [v1 packaging: project initialization](./v1-api/packaging/nodejs-init-project.md) - [v1 packaging: workspaces](./v1-api/packaging/nodejs-workspaces.md) @@ -44,3 +12,7 @@ - [v1 consuming: override packages](./v1-api/consuming/override.md) - [v1 integrating: lang2nix tool (pure)](./v1-api/integrating/integrate-lang2nix-pure.md) - [v1 integrating: lang2nix tool (code-gen/impure)](./v1-api/integrating/integrate-lang2nix-impure.md) +# Development Roundups +- [April - June 2022](./development-roundups/2022-april-june.md) +- [July - September 2022](./development-roundups/2022-july-september.md) + diff --git a/docs/src/api/generic-lock.md b/docs/src/api/generic-lock.md deleted file mode 100644 index cf7ec68e7d..0000000000 --- a/docs/src/api/generic-lock.md +++ /dev/null @@ -1,38 +0,0 @@ -# Generic lockfile - -The generic lockfile is interpreted from the fetchers and dependencies are loaded based on their type. - -When dream2nix reads project dependencies and translates them into -a dream2nix lockfile, it writes down the sources for all the -dependencies: so it can fetch them later during building. - -The generic lockfile is used always, either in memory or dumped to a file. - - -Below are some examples of the sources written in the dream-lock file -(at the end, under `sources`). - -```json -"symfony/process": { - "v6.1.3": { - "rev": "a6506e99cfad7059b1ab5cab395854a0a0c21292", - "type": "git", - "url": "https://github.com/symfony/process.git" - } -} -``` - -```json -"fast-glob": { - "3.2.11": { - "hash": "sha512-xrO3+1bxSo3ZVHAnqzyuewYT6aMFHRAd4Kcs92MAonjwQZLsK9d0SF1IyQ3k5PoirxTW0Oe/RqFgMQ6TcNE5Ew==", - "type": "http", - "url": "https://registry.npmjs.org/fast-glob/-/fast-glob-3.2.11.tgz" - } -} -``` - -The `type` field tells dream2nix what [fetchers](../intro/fetchers.md) to invoke to get this -source, the rest of the attributes are simply passed to the fetcher. - -You can `nix run` the `.resolve` attribute of a dream2nix package to save the generic lockfile for inspection. diff --git a/docs/src/contributing/00-declare-variables.sh b/docs/src/contributing/00-declare-variables.sh deleted file mode 100644 index 19b02b964f..0000000000 --- a/docs/src/contributing/00-declare-variables.sh +++ /dev/null @@ -1,10 +0,0 @@ -export dream2nix=$(realpath .) - -# define names for new modules (pick names matching to your subsystem) -export subsystem="my-subsystem" # example: nodejs -export pureTranslator="my-pure-translator" # example: package-lock -export impureTranslator="my-impure-translator" # example: package-json -export builder="my-builder" # pick `default` as name if not sure - -# define path to example flake -export myFlake="$dream2nix/examples/$subsystem" diff --git a/docs/src/contributing/01-initialize-templates.sh b/docs/src/contributing/01-initialize-templates.sh deleted file mode 100644 index d45b72c831..0000000000 --- a/docs/src/contributing/01-initialize-templates.sh +++ /dev/null @@ -1,7 +0,0 @@ -# initialize pure translator -mkdir -p $dream2nix/src/subsystems/$subsystem/translators/$pureTranslator -cp $dream2nix/src/templates/translators/pure.nix $dream2nix/src/subsystems/$subsystem/translators/$pureTranslator/default.nix - -# initialize builder -mkdir -p $dream2nix/src/subsystems/$subsystem/builders/$builder -cp $dream2nix/src/templates/builders/default.nix $dream2nix/src/subsystems/$subsystem/builders/$builder/default.nix diff --git a/docs/src/contributing/02-initialize-example-flake.sh b/docs/src/contributing/02-initialize-example-flake.sh deleted file mode 100644 index 566429ec83..0000000000 --- a/docs/src/contributing/02-initialize-example-flake.sh +++ /dev/null @@ -1,3 +0,0 @@ -# initialize example flake -mkdir -p $myFlake -cp $dream2nix/tests/integration/tests/contribute/my-flake.nix $myFlake/flake.nix diff --git a/docs/src/contributing/03-add-files-to-git.sh b/docs/src/contributing/03-add-files-to-git.sh deleted file mode 100644 index a58e0f3c3e..0000000000 --- a/docs/src/contributing/03-add-files-to-git.sh +++ /dev/null @@ -1 +0,0 @@ -git add . diff --git a/docs/src/contributing/04-test-example-flake.sh b/docs/src/contributing/04-test-example-flake.sh deleted file mode 100644 index 4120cfee8c..0000000000 --- a/docs/src/contributing/04-test-example-flake.sh +++ /dev/null @@ -1,8 +0,0 @@ -# inspect if your subsystem exports any package/devShell -nix flake show $myFlake --override-input dream2nix $dream2nix --show-trace - -# run your translator and dump the dream-lock.json for inspection -nix run $myFlake#default.resolve --override-input dream2nix $dream2nix --show-trace - -# test if the default package builds -nix build $myFlake#default --override-input dream2nix $dream2nix --show-trace diff --git a/docs/src/contributing/contributing.md b/docs/src/contributing/contributing.md deleted file mode 100644 index fa35a31068..0000000000 --- a/docs/src/contributing/contributing.md +++ /dev/null @@ -1,109 +0,0 @@ -# dream2nix contributors guide -This guide is for you if you plan to implement support for a new subsystem in dream2nix, like for example for a new programming language. - -If the ecosystem you are interested in is already supported by dream2nix, but you want to add support for a new type of lock-file format, this guide is still an interesting read in order to better understand the parts a dream2nix subsystem consists of. - -## Breakdown of a subsystem -A new subsystem in dream2nix is initialized by adding 3 files: - -- one translator module -- one builder module -- one example flake.nix for testing the subsystem - -It's also highly recommended to implement a discoverer module, so that projects of that subsystem can be detected automatically by dream2nix. This simplifies the UX. It won't be necessary anymore for the user to understand which ecosystem and which translator must be used in order to build packages from a certain source tree. - -## Translator Notes - -The task of a translator is to inspect a given source tree, parse some of the files, and extract information about a projects dependencies and how it must be built. - -In general there are 3 different types of translators. -No matter which type, all translators always produce the same output structure which is called `dream-lock`. -An example of this structure can be found [here](https://github.com/nix-community/dream2nix/blob/main/src/specifications/dream-lock-example.json). -There is also a [jsonschema specification](https://github.com/nix-community/dream2nix/blob/main/src/specifications/dream-lock-schema.json) for it. - -## Translator types -The different types of translators have the following properties: - -1. pure translator - - - returns the dream-lock as a nix attribute set - - translation logic is implemented in nix language only - - parsing of files and data extraction is all done during eval time - - does not invoke a build or read from any build output - -2. pure translator utilizing IFD (import from derivation) - - - returns the dream-lock as a nix attribute set - - a nix build is used in order to parse files and extract data - - translation logic can be implemented in arbitrary language - - the result is parsed back into nix from the build output - - downside: performance impact on evaluation time - -3. impure translator - - - returns the dream-lock by dumping a dream-lock.json file - - translator can be any executable program running independent of nix - - not constrained in any way (can do arbitrary network access etc.) - - downside: requires the user to run a command whenever dependencies got updated - -## Which translator type to start with? -When adding support for a new ecosystem/language, the following strategy usually works out: - -If there exists tooling within that ecosystem that can create some kind of lock file (with URLs + checksums), implement a pure translator for that lock file format first. - -After that, we might still need an impure translator for all the projects within that ecosystem that don't ship a lock-file. But given the fact that we already have one pure translator, all the impure translator needs to do is to run the tooling that creates the lock file and call out to the pure translator via `nix eval`. - -If the ecosystem does not have any kind of lock file format, then only an impure translator is needed. In this case it needs to be more complex and implement some kind of routine for retrieving all URL's and hashes of the dependencies, by, for example, downloading them all and hashing them. - -## Initializing the subsystm -To initialize a new subsystem, we will: - -- declare a few shell variables -- initialize a translator and a builder from templates -- initialize an example flake.nix to test the implementation - -### declare env variables -Navigate to your dream2nix checkout and execute: -```bash -{{#include ./00-declare-variables.sh}} -``` - -### initialize templates - -```bash -{{#include ./01-initialize-templates.sh}} -``` - -### initialize example flake.nix - -Initialize the flake from a template and edit it to reference the names of your subsystem correctly. -```bash -{{#include ./02-initialize-example-flake.sh}} -``` -Now edit the flake and ensure that `my-subsystem`, `my-pure-translator`, are replaced with the names defined earlier. - -### add new files to git -This is required, otherwise nix flakes won't see the new files. -```bash -{{#include ./03-add-files-to-git.sh}} -``` - -### test example flake -Always pass `--override-input dream2nix $dream2nix` in order to evaluate the example flake against your local checkout of dream2nix. - -In the following bash snippet, arguments containing a '`#`' symbol are wrongfully highlighted as comment but are in fact required parameters. - -Run all of the following commands now to ensure that all templates have been initialized correctly -```bash -{{#include ./04-test-example-flake.sh}} -``` - - -## Iterate on the subsystem -By default the templates implement a subsystem for `niv`. It reads niv's `./nix/sources.json` and builds a package for it containing the niv inputs. - -The output of this is not useful, but demonstrates how a dream2nix translator/builder works. - -You can now start modifying the builder/translator to implement the logic required by your subsystem. - -You can test your implementation by executing the `nix flake show`, `nix build`, `nix run` commands from the last step above. diff --git a/docs/src/define-targets.md b/docs/src/define-targets.md deleted file mode 100644 index 6772158698..0000000000 --- a/docs/src/define-targets.md +++ /dev/null @@ -1,50 +0,0 @@ -## Define target platform(s) -We have the flake setup, now we need to define the supported systems, -this is necessary because nix can do multi platform and cross-platform -builds so we need to tell it what can be built and where. - -There are 2 ways to do this, either with a `nix_systems` file, -or we can write the target platforms inline to our `flake.nix`. - -### nix_systems -We can create a `nix_systems` file with the current system: - -```command -> nix eval --impure --raw --expr 'builtins.currentSystem' > ./nix_systems -> git add ./nix_systems -``` - -The `nix_systems` file is simply a list of the supported systems, for example: -``` -x86_64-linux -``` - -Remember to add the file `./nix_systems` to git, or it won't be picked up by nix. -If you want to support more platforms later, just add more lines to that file. - -### inline -Alternatively, we can define the targets in the `flake.nix` like so: -```nix -{ - inputs.dream2nix.url = "github:nix-community/dream2nix"; - outputs = inp: - inp.dream2nix.lib.makeFlakeOutputs { - systems = ["x86_64-linux"]; # <- This line. - config.projectRoot = ./.; - source = ./.; - projects = ./projects.toml; - }; -} -``` - -This has the advantage of keeping all the configuration in a single file. - -## Populating projects.toml - -dream2nix also needs to know things about the project(s) at hand. -In the `flake.nix` file you can see it's expecting a `./projects.toml`. -The easiest way to create and populate this `./projects.toml` is with the helper function -```command -nix run github:nix-community/dream2nix#detect-projects . > projects.toml -git add projects.toml -``` diff --git a/docs/src/extending-dream2nix.md b/docs/src/extending-dream2nix.md deleted file mode 100644 index b14e8c343c..0000000000 --- a/docs/src/extending-dream2nix.md +++ /dev/null @@ -1,31 +0,0 @@ -# Extending dream2nix with external translators, builders etc. - -`dream2nix` uses the NixOS module system for it's internal components. -This means you can extend it using the module system. -This can be done while the framework is being `init`ialized. -To do this, you need to utilize the `config.modules` option of the dream2nix config. - -## Declaring `modules` - -`config.modules` is a configuration option that expects paths to module files. - -```nix -dream2nix.lib.init { - config.modules = [./nix/d2n/extras.nix]; -} -``` -this file can look like this: -```nix -{ config, ... }: -let - inherit (config) pkgs lib dlib; -in -{ - translators.example-translator = {/* translator attrs */}; - # you can declare modules using functions here - fetchers.ipfs = {/* fetcher attrs */}; -} -``` - -See the [`d2n-extended` example](https://github.com/nix-community/dream2nix/tree/main/examples/_d2n-extended) for an example on how to extend existing subsystems. -See the [`d2n-extended-new-subsystem` example](https://github.com/nix-community/dream2nix/tree/main/examples/_d2n-extended-new-subsystem) for an example on how to implement a new subsystem. diff --git a/docs/src/guides/getting-started-nodejs.md b/docs/src/guides/getting-started-nodejs.md deleted file mode 100644 index 7d4d80130a..0000000000 --- a/docs/src/guides/getting-started-nodejs.md +++ /dev/null @@ -1,266 +0,0 @@ -# Build your nodejs project with nix in 10 minutes or less - -{{#include ../warning.md}} - -This guide takes you step-by-step through setting up a nodejs -reproducible build and development environment using nix (the build -system) and dream2nix (bridge between external package managers and nix). - -This setup will allow for the same environment to be reproduced on -different machines and CI systems with high accuracy - thus avoiding -many pitfalls of distributed software development. - -## Outline -0. Install nix with flakes enabled -1. Navigate to your nodejs project -2. Initialize the dream2nix flake -3. Define target platform(s) -4. Explore the outputs -5. Build the project -6. Development shell -7. FAQ - -{{#include ../install-nix.md}} - -## Navigate to your nodejs project -For this guide we will use the fun -[`cowsay`](https://github.com/piuccio/cowsay) nodejs project. -It is simply a talking cow for your console. This project is a nodejs -port from the original perl version. -Feel free to use any other project, if you do and hit a roadblock, -please consult the [FAQ](#FAQ) at the end of this article for solutions -to some common issues. - -We start by cloning the project: -```command -> git clone https://github.com/piuccio/cowsay /tmp/my_project -> cd /tmp/my_project -``` - -## Initialize the dream2nix flake -We have our repository cloned and ready. Now we will create the flake. - -The flake is a standalone description of the project, it will define the -inputs and outputs of our project, the build steps - in this case -handled by dream2nix, and the development environment. -The flake is a fully standalone and complete configuration for nix to -build a software package. - -We use a dream2nix flake template: -```command -> nix flake init -t github:nix-community/dream2nix#simple -wrote: /tmp/my_project/flake.nix -``` -to create a `flake.nix`: -```nix -{ - inputs.dream2nix.url = "github:nix-community/dream2nix"; - outputs = inp: - inp.dream2nix.lib.makeFlakeOutputs { - systemsFromFile = ./nix_systems; - config.projectRoot = ./.; - source = ./.; - projects = ./projects.toml; - }; -} -``` -This file configures our build and development environment using the -dream2nix framework to bridge the nodejs ecosystem into nix. This let's -nix read and understand `package.json` and how to install and link -nodejs packages - to avoid duplication of dependency definitions and -build steps. - -{{#include ../define-targets.md}} - -## Explore the outputs -We have setup the flake, defined our target system(s), now we are ready -to use it. Let's start by listing out what is available to us (actual -output may be different, this is a shortened version): -```command -> nix flake show -warning: Git tree '/tmp/my_project' is dirty -warning: creating lock file '/tmp/my_project/flake.lock' -warning: Git tree '/tmp/my_project' is dirty -git+file:///tmp/my_project -β”œβ”€β”€β”€devShell -β”‚ └───x86_64-linux: development environment 'nix-shell' -β”œβ”€β”€β”€devShells -β”‚ └───x86_64-linux -β”‚ β”œβ”€β”€β”€cowsay: development environment 'nix-shell' -β”‚ └───default: development environment 'nix-shell' -└───packages - └───x86_64-linux - β”œβ”€β”€β”€cowsay: package 'cowsay-1.5.0' - β”œβ”€β”€β”€default: package 'cowsay-1.5.0' - └───resolveImpure: package 'resolve' -``` - -We can see that: -1. `warning: Git tree '/tmp/my_project' is dirty` -Our repository has uncommitted changes. -Nix uses git commit hashes to version build artifacts, -so this can result in some extra rebuilds. -Since we are just setting up the project now, this is alright. -1. `warning: creating lock file '/tmp/my_project/flake.lock'` -Our flake itself has an input (external dependency), the `dream2nix` -framework. When we first use the flake, like we just did, nix created a -lock file with the exact version of the input (and its inputs). Commit -this file to version control to ensure reproducible builds. -1. Finally, we see the outputs of our flake. -We see it outputs `packages` for the `x86_64-linux` systems: -the `cowsay` package (our nodejs project) and a `resolveImpure` package -(more about that in the next section). It also sets out `cowsay` package -as the `default` package of this flake. - -## Build the project -We have setup our flake for the nodejs project and identified the -output we want to build. - -To build the output, we run: -```command -> nix build .#cowsay -``` -(The `.` means the flake in the current directory, `#` is a special -character separating the flake name and the package name, and `cowsay` is -the name of the package we want to build. Since we want to `build`, nix -will look under `packages` first, and it knows our current platform -(`x86_64-linux` in this case), so it will actually build the output -`.#packages.x86_64-linux.cowsay`.) - -Since, `cowsay` is the `default` package, we could also simply run: -```command -> nix build -``` -To build the `.#packages.x86_64-linux.default` output. (In our case -these are the same.) - -This creates our `./result` directory with all our final build -artifacts. -```command -> ./result/bin/cowsay 'hello dream2nix' - _________________ -< hello dream2nix > - ----------------- - \ ^__^ - \ (oo)\_______ - (__)\ )\/\ - ||----w | - || || -``` - -Nix was able to build this project, because it has a `package-lock.json` -file pinning the exact dependency versions. If we did not have this -file, the `nix build` would fail with `error: unresolved impurities`. -```command -> git rm package-lock.json -> nix build .#cowsay -warning: Git tree '/tmp/my_project' is dirty -error: The nodejs package cowsay contains unresolved impurities. - Resolve by running the .resolve attribute of this derivation - or by resolving all impure projects by running the `resolveImpure` package -``` -We can fix this by generating a language specific lockfile -(`package-lock.json` or `yarn.lock` for nodejs), or let dream2nix -generate a universal `dream-lock.json`. -```command -> nix run .#resolveImpure -warning: Git tree '/tmp/my_project' is dirty -Resolving:: Name: cowsay; Subsystem: nodejs; relPath: -translating in temp dir: /tmp/tmp.SwBFt0WcH4 - -up to date, audited 172 packages in 9s - -57 packages are looking for funding - run `npm fund` for details - -3 high severity vulnerabilities - -To address all issues (including breaking changes), run: - npm audit fix --force - -Run `npm audit` for details. -===[SUCCESS]===(dream2nix-packages/cowsay/dream-lock.json)=== -adding file to git: dream2nix-packages/cowsay/dream-lock.json -``` -This runs `npm` behind the scenes and resolves dependencies for all -packages inside the flake. - -There is no difference between using an external lockfile or -`dream-lock.json`, all they do is pin dependency version and are -completely interchangeable. - -## Development shell -We were able to build our nodejs project with nix, however our build -artifacts under `./result` are read-only and we do not have `node` and -`npm` in `PATH`. To be able to work in this project we will need those. - -Nix provides us with `devShells` for exactly this. -```command -> nix flake show -warning: Git tree '/tmp/my_project' is dirty -warning: creating lock file '/tmp/my_project/flake.lock' -warning: Git tree '/tmp/my_project' is dirty -git+file:///tmp/my_project -β”œβ”€β”€β”€devShell -β”‚ └───x86_64-linux: development environment 'nix-shell' -β”œβ”€β”€β”€devShells -β”‚ └───x86_64-linux -β”‚ β”œβ”€β”€β”€cowsay: development environment 'nix-shell' -β”‚ └───default: development environment 'nix-shell' -└───packages - └───x86_64-linux - β”œβ”€β”€β”€cowsay: package 'cowsay-1.5.0' - β”œβ”€β”€β”€default: package 'cowsay-1.5.0' - └───resolveImpure: package 'resolve' -``` -When we enter the `cowsay` development shell, we will get `node` in our -`PATH`, together with all the binaries from our dependencies packages. -And nix will copy over `node_modules` for us to save us from having to -`npm install` everything over again. -To get in the shell, simply run: -```command -> nix develop -c $SHELL -``` -(The `-c $SHELL` part is only necessary if you use a different shell than bash -and would like to bring that shell with you into the dev environment.) - -From here on it's the same as using a normal installation of nodejs. -However, if we do imperative changes to `node_modules` and later -re-enter the nix development shell, nix will overwrite the -`node_modules` with the pinned versions of the dependencies from our -lockfile. - -## FAQ - -### Refusing to overwrite existing file on `flake init`. - -When initializing a flake it needs to write some files, if these already -exists, the initialization will fail. Since our repository is under -version control, we can delete the conflicting files, let `flake init` -create them and then check the diff and merge the changes manually. - -### Getting status of `flake.nix`: no such file or directory. - -The flake build does not happen inside the directory. Nix copies your -repository to a temporary location and builds there; only files under -version control are used. To resolve this run `git add flake.nix` and -all other missing files. - -### Warning: Git tree is dirty - -This is just a warning, nix is using the git revision for build artifact -versioning. Having a dirty git tree - meaning uncommitted changes - can -lead to some extra rebuilds, for simple projects this should not be a -major concern. - -### error: The package contains unresolved impurities. Resolve all impure projects by running the `resolveImpure` package. - -This happens when dream2nix cannot resolve exact package versions. We -can define a dependency like `something@^2.1`, but it is not obvious if -we actually want `2.1.1` or `2.1.2` or maybe `2.1.1-alpha`. -There are 2 ways to resolve this error: either by running -`nix run .#resolveImpure` and letting dream2nix resolve the most -up-to-date versions of all dependencies, or using an external package -manager to generate a lock file, which can be later read by dream2nix. -(In case of Node.js, both `package-lock.json` and `yarn.lock` are -supported.) diff --git a/docs/src/guides/getting-started-python.md b/docs/src/guides/getting-started-python.md deleted file mode 100644 index d02628299f..0000000000 --- a/docs/src/guides/getting-started-python.md +++ /dev/null @@ -1,102 +0,0 @@ -# Build your python project with nix in 10 minutes - -{{#include ../warning.md}} - -This guide walks you through the process of setting up nix for your project using dream2nix. This will allow your project's build and dev-environment to be reproduced by machines of other developers or CI systems with high accuracy. - -## Outline - -0. Install nix -1. Navigate to your python project -2. Initialize the dream2nix flake -3. Define target platform(s) -4. List the available packages -5. Build the project -6. Create a development shell -7. Resolve impurities - -{{#include ../install-nix.md}} - -## Navigate to your python project -In this example I will clone the python project [`httpie`](https://github.com/httpie/httpie) to `/tmp/my_project` as an example. -```command -> git clone https://github.com/httpie/httpie /tmp/my_project -> cd /tmp/my_project -``` - -## Initialize the dream2nix flake.nix -```command -> nix flake init -t github:nix-community/dream2nix#simple -wrote: /tmp/my_project/flake.nix -``` -Great, this created a new file `flake.nix` which is like a recipe that tells nix how to build our python project or how to assemble a development environment for it. -By modifying this file, we can tweak settings and change the way our package gets built by nix. But for now we just go with the defaults. - -{{#include ../define-targets.md}} - -## List the available packages -Let's get an overview of what the `flake.nix` allows us to do with our project. -```command -> nix flake show -warning: Git tree '/tmp/my_project' is dirty -warning: creating lock file '/tmp/my_project/flake.lock' -warning: Git tree '/tmp/my_project' is dirty -git+file:///tmp/my_project -└───packages - └───x86_64-linux - β”œβ”€β”€β”€main: package 'main' - └───resolveImpure: package 'resolve' -``` - -What we can observe here: -1. ```warning: Git tree '/tmp/my_project' is dirty``` -Nix warns us that the current git repo has uncommited changes. Thats fine, because we like to experiment for now. This warning will go away as soon as we commit our changes. -1. `warning: creating lock file '/tmp/my_project/flake.lock'` -Our flake.nix imported external libraries. The versions of these libraries have now been locked inside a new file `flake.lock`. We should later commit this file to the repo, in order to allow others to reproduce our build exactly. -1. - ``` - git+file:///tmp/my_project - └───packages - └───x86_64-linux - β”œβ”€β”€β”€main: package 'main' - └───resolveImpure: package 'resolve' - ``` - Similar like a .json file defines a structure of data, our flake.nix defines a structure of `nix attributes` which are things that we can build or run with nix. - We can see that it contains packages for my current platform `x86_64-linux`. - - The packages which we can see here is my python package and a package called `resolveImpure`, which is a special package provided by dream2nix which we will learn more about later. - -## Build the project -Let's try building our project. -If you get an error about `unresolved impurities`, see [Resolve Impurities](#resolve-impurities) -```command -> nix build .#main -``` -Congratulations, your build artifacts will now be accessible via the `./result` directory. If your project contains executables, you can run these via `./result/bin/executable-name`. -If you want to develop on your python project, see [Create a development shell](#create-a-development-shell) - -## Create a development shell -Nix can provide you with a development shell containing all your project's dependencies. -First, ensure that your project [is resolved](#resolve-impurities), then execute the following command. -```command -> nix develop -c $SHELL -``` -The `-c $SHELL` part is only necessary if you use a different shell than bash and would like to bring that shell with you into the dev environment. - -## Resolve impurities -If you try to build, you might run into the following error. -```command -> nix build .#main -error: The python package main contains unresolved impurities. - Resolve by running the .resolve attribute of this derivation - or by resolving all impure projects by running the `resolveImpure` package -``` -Oops. It seems like our project does not contain enough information for dream2nix to construct a reproducible build. But this is not a problem as we can fix this by using the `resolveImpure` package that dream2nix provides. -```command -> nix run .#resolveImpure -... -adding file to git: dream2nix-packages/main/dream-lock.json -``` -Fine, that created a new file `dream-lock.json` which is a lock file specifically for our python project. If we later add any dependencies, we will have to re-run `resolveImpure` to update this lock file. - -Now everything should be ready to [Build the Project](#build-the-project) diff --git a/docs/src/install-nix.md b/docs/src/install-nix.md deleted file mode 100644 index 4464171e3b..0000000000 --- a/docs/src/install-nix.md +++ /dev/null @@ -1,13 +0,0 @@ -## Install nix -If you don't have nix already, check out [nixos.org/download.html](https://nixos.org/download.html) on how to install it. - -### Enable the nix flakes feature -For internal dependency management dream2nix requires the experimental nix feature `flakes` being enabled. -``` -export NIX_CONFIG="extras-experimental-features = flakes nix-command" -``` - -If you find yourself using dream2nix regularly, you can permanently save these settings by adding the following line to your `/etc/nix/nix.conf`: -``` -experimental-features = flakes nix-command -``` diff --git a/docs/src/intro.md b/docs/src/intro.md index 5cf9f72e9f..65ab65cf8c 100644 --- a/docs/src/intro.md +++ b/docs/src/intro.md @@ -1,20 +1,33 @@ -# dream2nix -Automate reproducible packaging for various language ecosystems +

+ + + + dream2nix - A framework for automated nix packaging + +
+ Automate reproducible packaging for various language ecosystems +
+ Documentation | + Examples +

-{{#include ./warning.md}} +!!! Warning: dream2nix is unstable software. While simple UX is one of our main focus points, the APIs are still under development. Do expect changes that will break your setup. -{{#include ./funding.md}} +### Funding + +This project was funded through the [NGI Assure](https://nlnet.nl/assure) Fund, a fund established by [NLnet](https://nlnet.nl/) with financial support from the European Commission's [Next Generation Internet](https://ngi.eu/) programme, under the aegis of DG Communications Networks, Content and Technology under grant agreement No 957073. **Applications are still open, you can [apply today](https://nlnet.nl/propose)**. + +If your organization wants to support the project with extra funding in order to add support for more languages or new features, please contact one of the maintainers. + +## Goals dream2nix focuses on the following aspects: - Modularity - Customizability - Maintainability -- Nixpkgs Compatibility, by not enforcing [IFD (import from derivation)][glossary] -- Code de-duplication across 2nix converters -- Code de-duplication in nixpkgs -- Risk-free opt-in aggregated fetching (larger [FODs][glossary], less checksums) -- Common UI across 2nix converters +- Code de-duplication across 2nix solutions +- Common UI across 2nix solutions - Reduce effort to develop new 2nix solutions - Exploration and adoption of new nix features - Simplified updating of packages @@ -23,60 +36,15 @@ The goal of this project is to create a standardized, generic, modular framework The intention is to integrate many existing 2nix converters into this framework, thereby improving many of the previously named aspects and providing a unified UX for all 2nix solutions. +### Documentation -### Modularity: -The following phases which are generic to basically all existing 2nix solutions: - - parsing project metadata - - resolving/locking dependencies (not always required) - - fetching sources - - building/installing packages - -... should be separated from each other with well defined interfaces. - -This will allow for free composition of different approaches for these phases. -The user should be able to freely choose between: - - input metadata formats (eg. lock file formats) - - metadata fetching/translation strategies: IFD vs. in-tree - - source fetching strategies: granular fetching vs fetching via single large FOD to minimize expression file size - - installation strategies: build dependencies individually vs inside a single derivation. - -### Customizability -Every Phase mentioned in the previous section should be customizable at a high degree via override functions. Practical examples: - - Inject extra requirements/dependencies - - fetch sources from alternative locations - - replace or modify sources - - customize the build/installation procedure - -### Maintainability -Due to the modular architecture with strict interfaces, contributors can add support for new lock-file formats or new strategies for fetching, building, installing more easily. - -### Compatibility -Depending on where the nix code is used, different approaches are desired or discouraged. While IFD might be desired for some out of tree projects to achieve simplified UX, it is strictly prohibited in nixpkgs due to nix/hydra limitations. -All solutions which follow the dream2nix specification will be compatible with both approaches without having to re-invent the tool. - -### Code de-duplication -Common problems that apply to many 2nix solutions can be solved once by the framework. Examples: - - handling cyclic dependencies - - handling sources from various origins (http, git, local, ...) - - generate nixpkgs/hydra friendly output (no IFD) - - good user interface - -### Code de-duplication in nixpkgs -Essential components like package update scripts or fetching and override logic are provided by the dream2nix framework and are stored only once in the source tree instead of several times. - -### Risk free opt-in FOD fetching -Optionally, to save more storage space, individual hashes for source can be omitted and a single large FOD used instead. -Due to a unified minimalistic fetching layer the risk of FOD hash breakages should be very low. - -### Common UI across many 2nix solutions -2nix solutions which follow the dream2nix framework will have a unified UI for workflows like project initialization or code generation. This will allow quicker onboarding of new users by providing familiar workflows across different build systems. +[πŸ‘‰ To the docs](https://nix-community.github.io/dream2nix) -### Reduced effort to develop new 2nix solutions -Since the framework already solves common problems and provides an interface for integrating new build systems, developers will have an easier time creating their next 2nix solution. +### Presentations -### Further reading +- [πŸ‘‰ 2021: Original dream2nix presentation](https://www.youtube.com/watch?v=jqCfHMvCsfQ) - Examples are outdated: +- [πŸ‘‰ 2023: dream2nix based on drv-parts](hhttps://www.youtube.com/watch?v=AsCvRZukX0E) - Examples are outdated: -- [Architectural Considerations](./intro/architecture.md) -- [Potential impact on nixpkgs](./intro/nixpkgs-improvements.md) +### Community -[glossary]: https://nixos.wiki/wiki/Glossary "glossary" +[πŸ‘‰ matrix chat room](https://matrix.to/#/#dream2nix:nixos.org) diff --git a/docs/src/intro/architecture.md b/docs/src/intro/architecture.md deleted file mode 100644 index f9f1b73015..0000000000 --- a/docs/src/intro/architecture.md +++ /dev/null @@ -1,127 +0,0 @@ -# Architecture - -The general architecture should consist of these components: -`Input -> Translation -> Generic Lock -> Fetching -> Building` - -``` -β”Œβ”€β”€β”€β”€β”€β”€β”€β” -β”‚ Input │◄── Arbitrary -β””β”€β”€β”€β”€β”¬β”€β”€β”˜ URLs + Metadata containing Build instructions - β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” in standardized minimalistic form (json) - └──►│Translatorβ”‚ β”‚ - β””β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”˜ β–Ό - β–² β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” - β”‚ └──►│Generic Lockβ”‚ - β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”˜ - - pure-nix β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β” - - IFD / recursive-nix β”œβ”€β”€β–Ίβ”‚Fetcher │◄── Same across all - - impure (external) β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”˜ languages/frameworks - β”‚ β–Ό - β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β” - └──►│Builder │◄── Reads extra metadata - β””β”€β”€β”€β”€β”€β”€β”€β”€β”˜ from generic lock -``` - -Input: -- can consist of: - - requirement constraints - - requirement files - - lock-files - - project's source tree - -Translator: - - read input and generate generic lock format containing: - - URLs + hashes of sources - - metadata for building - - different strategies can be used: - - `pure-nix`: translate input by using the nix language only - - `IFD/recursive`: translate using a nix build - - `external`: translate using an external tool which resolves against an online package index - - for more information about translators and how nixpkgs compatibility is guaranteed, check [Translators](./translators.md) - -Generic Lock (standardized format): - - Produced by `Translator`. Contains URLs + hashes for sources and metadata relevant for building. - - The contained format for sources and dependency relations is independent of the build system. Fetching works always the same. - - The metadata also contains build system specific attributes as individual approaches are required here. A specific builder for the individual build system will later read this metadata and transform it into nix derivations. - - It is not relevant which steps/strategies have been taken to create this lock. From this point on, there are no impurities. This format will contain everything necessary for a fully reproducible build. - - This format can always be put into nixpkgs, not requiring any IFD (given the nix code for the builder exists within nixpkgs). - - In case of a pure-nix translator, the generic lock data can be generated on the fly and passed directly to the builder, preventing unnecessary usage of IFD. - -Fetcher: - - Since a generic lock was produced in the previous step, the fetching layer can be the same across all build systems. - -Builder: - - Receives sources from fetcher and metadata produced by the translator. - - The builder transforms the metadata into nix derivation(s). - - Strictly separating the builder from previous phases allows: - - switching between different build strategies or upgrading the builder without having to re-run the translator each time. - - reducing code duplication if a project contains multiple packages built via dream2nix. - - -### Example (walk through the phases) -#### python project with poetry.lock -As an example we package a python project that uses poetry for dependency management. -Poetry uses `pyproject.toml` and `poetry.lock` to lock dependencies -- Input: pyproject.toml, poetry.lock (toml) -- Translator: written in pure nix, reading the toml input and generating the generic lock format -- Generic Lock (for explanatory purposes dumped to json and commented): - ```json - { - // generic lock format version - "version": 1, - - // format for sources is always the same (not specific to python) - "sources": { - "requests": { - "type": "tarball", - "url": "https://download.pypi.org/requests/2.28.0", - "hash": "deadbeefdeadbeefdeadbeefdeadbeefdeadbeef", - }, - "certifi": { - "type": "github", - "owner": "certifi", - "repo": "python-certifi", - "hash": "deadbeefdeadbeefdeadbeefdeadbeefdeadbeef" - } - }, - - // generic metadata (not specific to python) - "_generic": { - - // this indicates which builder must be used - "subsystem": "python", - - // translator which generated this file - // (not relevant for building) - "producedBy": "translator-poetry-1", - - // dependency graph of the packages - "dependencies": { - "requests": [ - "certifi" - ] - } - }, - - // all fields inside 'subsystem' are specific to - // the selected subsystem (python) - "_subsystem": { - - // tell the python builder how the inputs must be handled - "sourceFormats": { - "requests": "sdist", // triggers build instructions for sdist - "certifi": "wheel" // triggers build instructions for wheel - } - } - } - ``` -- This lock data can now either: - - be dumped to a .json file and committed to a repo - - passed directly to the fetching/building layer -- the fetcher will only read the sources section and translate it to standard fetcher calls. -- the building layer will read the "subsystem" attribute and select the python builder for building. -- the python builder will read all information from "subsystem" and translate the data to a final derivation. - -Notes on IFD, FOD and code generation: -- No matter which type of translator is used, it is always possible to export the generic lock to a file, which can later be evaluated without using IFD or FOD, similar to current nix code generators, just with a standardized format. -- If the translator supports IFD or is written in pure nix, it is optional to the user to skip exporting the generic lock and instead evaluate everything on the fly. diff --git a/docs/src/intro/fetchers.md b/docs/src/intro/fetchers.md deleted file mode 100644 index d842f683ea..0000000000 --- a/docs/src/intro/fetchers.md +++ /dev/null @@ -1,118 +0,0 @@ -# Fetchers - -This section describes the available source code fetchers. A fetcher is responsible for loading sources defined in the generic lockfile described earlier. - -## Fetcher structure - -Here is the implementation of the simple `path` fetcher: -```nix -{{#include ../../../src/fetchers/path/default.nix}} -``` - -It defines `inputs` which are the required arguments it expects to -receive when invoked. -Under `outputs` is a function that returns an object with `calcHash` and -`fetched` function attributes. - -- `calcHash`: receives an algorithm and outputs the source hash, it - is used while generating the dream-lock file. -- `fetched`: receives a `hash` and returns the fetched source. - -It is important to remember that nix enforces reproducibility, to -download something from nix we need to be sure it is exactly what we -expect: usually enforced by comparing hashes. And you will see that -almost all fetchers need to be supplied with a `hash` of the source being -fetched. - -## Implemented fetchers - -The fetcher name corresponds to the source `type` used to invoke it. - -You can inspect the implementation of all the fetchers -[here](https://github.com/nix-community/dream2nix/tree/main/src/fetchers). - -### path - -Fetches from a path. Useful when we already have the source. - -Inputs: -- path - -_Does not require a `hash` since nix can check integrity of paths._ - -### http - -Fetches from a http address. - -Inputs: -- url -- hash - -### git - -Fetches from a git commit. - -Inputs: -- url -- rev - -_Does not require a `hash` since nix can use the `rev` to check for -integrity instead._ - -### github - -Fetches from a GitHub repository commit. - -Inputs: -- owner -- repo -- rev -- hash - -### gitlab - -Fetches from a GitLab repository commit. - -Inputs: -- owner -- repo -- rev -- hash - -### npm - -Fetches from npm registry. - -Inputs: -- pname -- version -- hash - -### pypi-sdist - -Fetches sources distributions ("sdists") from pypi registry. - -Inputs: -- pname -- version -- hash - -### pypi-wheel - -Fetches wheels from pypi registry. - -Inputs: -- filename -- pname -- version -- hash - - -### crates-io - -Fetches from crates.io registry. - -Inputs: -- pname -- version -- hash diff --git a/docs/src/intro/indexers.md b/docs/src/intro/indexers.md deleted file mode 100644 index c103c347fd..0000000000 --- a/docs/src/intro/indexers.md +++ /dev/null @@ -1,45 +0,0 @@ -# Indexers - -Indexers are programs that can query a package repository (think of npm, or crates.io) for package information. -This information mainly consists of the package name, the package version, and anything extra that might be useful / needed to fetch / translate it. -The information is stored as JSON (see below). - -## Indexer inputs - -Indexers take input as a path to a JSON file, which contains custom arguments for the indexer. -A common attribute for these inputs across indexers are `outputFile`, which should be the path to output the generated index to. -Indexers can vary in functionality, so these JSON inputs should be specified under `src/specifications/indexers/`. - -## Indexer outputs - -Indexers should output their generated index to where `outputFile` specifies. -This index should simply be a list of project specifications in JSON. Example: - -```json -[ - { - "name": "execa", - "version": "6.1.0", - "translator": ["npm"] - }, - { - "name": "meow", - "version": "10.1.3", - "translator": ["npm"] - }, - { - "name": "npm-run", - "version": "5.0.1", - "translator": ["npm"] - } -] -``` - -## Current indexers - -Following are the current indexers implemented in dream2nix: - -- **crates-io-simple**: crates.io indexer -- **crates-io**: crates.io indexer written in rust with more options -- **libraries-io**: multi ecosystem indexer utilizing libraries.io (requires API key) -- **npm**: simple indexer using npm's registry diff --git a/docs/src/intro/nixpkgs-improvements.md b/docs/src/intro/nixpkgs-improvements.md deleted file mode 100644 index b5b5d41de5..0000000000 --- a/docs/src/intro/nixpkgs-improvements.md +++ /dev/null @@ -1,92 +0,0 @@ -## List of problems which currently exist in nixpkgs - -### Generated Code Size/Duplication - -#### Problem - -- large .nix files containing auto generated code for fetching sources (example: nodejs) -- many duplicated .nix files containing build logic - -#### Solution - -- dream2nix minimizes the amount of generated nix code, as most of the logic required to build a package resides in the framework and therefore is not duplicated across individual packages. - -- If the upstream lock file format can be interpreted with pure nix and is present at evaluation time, then generating any intermediary code can be omitted. - -- Once any kind of recursive nix (IFD, recursive-nix, RFC-92) is enabled in nixpkgs, dream2nix will utilize it and eliminate the requirement of generating nix code or storing upsteam lock files - -### Update Scripts Duplication/Complexity - -#### Problem - -- update scripts are largely duplicated -- update scripts are complex - -#### Solution - -- storing `update.sh` scripts alongside packages will not be necessary anymore. dream2nix can generate update procedures on the fly by reading the package declaration. -- The UI for updating packages is the same across all languages/frameworks - -### Fetching / Caching issues (large FODs) - -#### Problem - -- non-reproducible large FOD fetchers (example: rust) -- updating FODs is not risk free (forget to update hash) -- bad caching properties due to large FODs - -#### Solution - -- the translators of dream2nix always produce a clear list of URLs to fetch -- large-FOD fetching is not necessary and never enforced -- large-FOD fetching can be used optionally to reduce amount of hashes to be stored -- even if large-FOD fetching is used, it won't have any of the known reproducibility issues, since dream2nix does never make use of upstream toolchain for fetching and potentially impure operations like dependency resolution are never done inside an FOD. -- updating hashes of FODs is done via dream2nix CLI, which ensures that the correct hashes are in place -- As the use of large-FOD fetching is not necessary and therefore minimized, dependencies are cached on an individual basis and shared between packages. - -### Update Workflows - -#### Problem - -- package update workflows can be complicated -- package update workflows vary significantly depending on the language/framework - -#### Solution - -- the workflow for updating packages will be unified and largely independent of the underlying language/framework. - -### Merge Conflicts for shared dependencies - -#### Problem - -- Due to how shared dependencies are managed, merge conflicts are likely (example: global node-packages.nix) - -#### Solution - -- Having a central set of shared dependencies can make sense to reduce the code size of nixpkgs, load on hydra+cache. -- To eliminate merge conflicts, the global package set can be maintained via a two stage process. Individual package maintainers can manage their dependencies independently. Once every staging cycle, common dependencies can be found via graph analysis and moved into a global package set. -- The total amount of dependency versions used can also be minimized by re-running the resolver, prioritizing dependencies from the global set of common packages. - -### Customizability / Overriding - -#### Problem - -- Capabilities vary depending on the underlying generator/translator. -- UI is different depending on the underlying generator/translator. - -#### Solution - -- dream2nix provides good interfaces for customizability which are unified as much as possible independently from the underlying subsystems. - -### Inefficient/Slow Innovation - -#### Problem - -- Design issues (FOD-impurity, Maintainability, etc.) cannot be fixed easily and lead to long term suffering of maintainers. -- Innovation often happens on individual tools and are not adopted ecosystem wide -- New nix features will not be easily adopted as this will require updating many individual tools. - -#### Solution - -- Since dream2nix centrally handles many core elements of packaging like different strategies for fetching and building, it is much easier to fix problems at large scale and apply new innovations to all underlysing subsystems at once. -- Experimenting with and adding support for new nix features will be easier as the framework offers better abstractions than existing 2nix converters and allows adding/modifying strategies more easily. diff --git a/docs/src/intro/override-system.md b/docs/src/intro/override-system.md deleted file mode 100644 index 851a822ac2..0000000000 --- a/docs/src/intro/override-system.md +++ /dev/null @@ -1,160 +0,0 @@ -The override system plays an important role when packaging software with dream2nix. Overrides are the only way to express package specific logic in dream2nix. This serves the purpose of strictly separating: -``` - - generic logic (builders) - - specific logic (overrides) - - data (dream-lock.json) -``` - -To optimize for scalable workflows, the structure of dream2nix overrides differs from the ones seen in other projects. -dream2nix overrides have the following properties: - - **referenceable**: each override is assigned to a key through which it can be referenced. This allows for better inspection, selective debugging, replacing, etc. - - **conditional**: each override can declare a condition, so that the override only applies when the condition evaluates positively. - - **attribute-oriented**: The relevant parameters are attributes, not override functions. dream2nix will automatically figure out which underlying function (eg. override, overrideAttrs, ...) needs to be called to update each given attribute. The user is not confronted with this by default. - -Each subsystem in dream2nix like `nodejs` or `python` manages its overrides in a separate directory to avoid package name collisions. - -dream2nix supports packaging different versions of the same package within one repository. Therefore conditions are used to make overrides apply only to certain package versions. - -Currently a collection of overrides is maintained at [dream2nix/overrides](https://github.com/nix-community/dream2nix/tree/main/overrides) - -## General override system - -values can either be declared directly via - -```nix -# "${pname}" = { -# "${overrideName}" = { -# ... -# `attrName` will get overriden with `newValue` -# ${attrName} = newValue; -# ... -# }; -# }; -# -``` - -or via function that takes the `oldAttrs` and returns `newAttrs` depending on the old ones. - -```nix -# -# "${pname}" = { -# "${overrideName}" = { -# ... -# overrideAttrs = oldAttrs: { -# ${attrName} = ...; -# }; -# ... -# }; -# }; -``` - -## Overview of `attrNames` - -The available values depend on the subsystem -But at least all values of `pkgs.mkDerivation` are available on every subsystem - -```nix - -# some internal attributes -# if that attribute is true the override will apply -# e.g. _condition = satisfiesSemver "^5.0.0"; -_condition - - -# attributes of the nodejs subsystem -dependenciesJson -electronHeaders -nodeDeps -nodeSources -packageName -installMethod -electronAppDir -runBuild -linkBins -installDeps -buildScript - -# python script to modify some metadata to support installation -# (see comments below on d2nPatch) -fixPackage - -# attibutes of mkDerivation (also found in the nix manual) -nativeBuildInputs -buildInputs -src -configurePhase -buildPhase -installPhase -patches -... - -``` - -# Example for nodejs overrides - -```nix -{ - # The name of a package. - # Contains all overrides which can apply to the package `enhanced-resolve` - enhanced-resolve = { - - # first override for enhanced-resolve named `preserve-symlinks-v4` - preserve-symlinks-v4 = { - - # override will apply for packages with major version 4 - _condition = satisfiesSemver "^4.0.0"; - - # this statement replaces exisiting patches - # (for appending see next example) - patches = [ - ./enhanced-resolve/npm-preserve-symlinks-v4.patch - ]; - - }; - - # second override for enhanced-resolve - preserve-symlinks-v5 = { - - # override will apply for packages with major version 5 - _condition = satisfiesSemver "^5.0.0"; - - # this statement adds a patch to the exsiting list of patches - patches = old: old ++ [ - ./enhanced-resolve/npm-preserve-symlinks-v5.patch - ]; - }; - - }; - - # another package name - webpack = { - # overrides for webpack - }; -} -``` - -# Example for PHP flake override - -This example overrides `prePatch` for the `ml/iri` package to drop the -unsupported `target-dir` attribute from composer.json: - -``` -{ - inputs.dream2nix.url = "github:nix-community/dream2nix"; - - outputs = inp: - inp.dream2nix.lib.makeFlakeOutputs { - systems = ["x86_64-linux"]; - config.projectRoot = ./.; - source = ./.; - projects = ./projects.toml; - packageOverrides = { - "^ml.iri.*".updated.overrideAttrs = old: { - prePatch = '' - cat composer.json | grep -v target-dir | sponge composer.json - ''; - }; - }; - }; -} -``` diff --git a/docs/src/intro/translators.md b/docs/src/intro/translators.md deleted file mode 100644 index 7585079af2..0000000000 --- a/docs/src/intro/translators.md +++ /dev/null @@ -1,45 +0,0 @@ -# Translators - -This document classifies different methods for translating requirement/lock -files to the generic lock format and describes how the resulting packages -can be integrated into nix builds inside and outside of nixpkgs. - - -## pure-nix (preferred) -Suitable if: -- the input contains information like URLs and hashes -- nix supports the hashing algorithm -- the input can be processed with the nix language directly - -Usage outside of nixpkgs: -- The input + translator are enough, not requiring any pre-processing. - -Usage inside nixpkgs: -- The input + translator are enough, not requiring any pre-processing. - -## IFD/recursive (compatible with import from derivation or recursive nix) -Suitable if: -- the input contains information like URLs and hashes -- nix understands the hashing algorithm -- to process the input, a nix build is required, because for example: - - the format cannot be parsed with the nix language (yaml etc.) - - processing the input is too complex and therefore inefficient in nix language - -Usage outside of nixpkgs: -- The input + translator are enough. The generic lock file is generated via IFD - -Usage inside nixpkgs: -- generic lock file must be pre-generated using dream2nix cli - -## impure (running outside of nix build) -Suitable if: -- the input is missing URLs or hashes -- the method used to process the input contains impurities, like for example: - - queries to an online index with varying responses - - packages must be downloaded to discover important meta data like dependencies. - -Usage outside of nixpkgs: -- generic lock file must be pre-generated using dream2nix cli - -Usage inside nixpkgs: -- generic lock file must be pre-generated using dream2nix cli diff --git a/docs/src/subsystems/haskell.md b/docs/src/subsystems/haskell.md deleted file mode 100644 index aabca4d2c0..0000000000 --- a/docs/src/subsystems/haskell.md +++ /dev/null @@ -1,42 +0,0 @@ -# Haskell subsystem - -This section documents the Haskell subsystem. - -Get involved: -- [understand dream2nix architecture](../intro/architectural-considerations.md#architecture) -- [understand dream2nix translator types](../intro/translators.md) -- [try out examples](https://github.com/nix-community/dream2nix/tree/main/examples) -- [find Issues](https://github.com/nix-community/dream2nix/issues?q=is%3Aissue+is%3Aopen+label%3Ahaskell) -- [find TODOs](https://sourcegraph.com/search?q=context:global+repo:%5Egithub%5C.com/nix-community/dream2nix%24+file:haskell/+TODO&patternType=literal) - -## Status -The Haskell subsystems is currently work in progress. -It is in a state where it can be used, but will potentially fail on many projects, because some important features are missing. -Currently we have a builder for haskell and several translators (see table below). - -The main elements which are missing: - -- a pure translator for cabal.project.freeze -- an impure cabal translator, that works with any cabal based project by generating a cabal.project.freeze file. This should be fairly simple. Once we have the pure cabal.project.freeze translator, the impure translator can just execute `cabal freeze` and then call out to the existing pure translator. -- detecting which GHC version must be used for a project -- source different GHC versions. For now we should probably just support all ghc versions from nixpkgs because those are cached on nixos.org. If the translated project requires a different ghc version, the user should be allowed to override this, or pick a different nixpkgs. -- add an indexer for stackage/hackage. That should allow us to build a repository with the most common haskell libraries equivalent to dream2nix-crates-io or dream2nix-npm. This will allow us to improve our builders and translators based on the errors we get. - - -## Examples -flake.nix -```nix -{{#include ../../../examples/haskell_stack-lock/flake.nix}} -``` - -see more examples under: [/examples](https://github.com/nix-community/dream2nix/tree/main/examples) - -## Translators - -The source code of the implementations can be found [here](https://github.com/nix-community/dream2nix/tree/main/src/subsystems/haskell/translators). - - -**translator** | stack-lock | cabal-plan | cabal-freeze | ------------|-----------------|-------------|----------------------| -**file name** | stack.yaml.lock | plan.json | cabal.project.freeze | -**status** | implemented | implemented | missing | diff --git a/docs/src/subsystems/node.md b/docs/src/subsystems/node.md deleted file mode 100644 index 1e50d294c4..0000000000 --- a/docs/src/subsystems/node.md +++ /dev/null @@ -1,124 +0,0 @@ -# Node.js subsystem - -This section documents the Node.js subsystem. - -## Example - -> ./flake.nix - -```nix -{{#include ../../../examples/nodejs_eslint/flake.nix}} -``` - -> ./projects.toml - -```toml -{{#include ../../../examples/nodejs_eslint/projects.toml}} -``` - -## Translators - -### package-lock (pure) - -Translates `package-lock.json` into a dream2nix lockfile. - -### package-json (impure) - -Resolves dependencies from `package.json` using `npm` to generate a -`package-lock.json`, then uses `package-lock` translator to generate the -dream2nix lockfile. - -### yarn-lock (pure) - -Translates `yarn.lock` into a dream2nix lockfile. - -## Builders - -### granular (pure) (default) - -Builds all the dependencies in isolation, moving upwards to the top -package. -At the end copies over all dependencies into `node_modules` and writes -symlinks for the bins into `node_modules/.bin`. - -### strict (pure + best compatibility) (experimental) - -Works almost the same as the granular builder. Not bulletproof stable yet. -Recommended: Try it out, it should work better than the current default builder, but is not yet released as default. - -Features 🌈 - -- Fully npm compatible -- No Patches / Overrides required (if "installMethod = copy") - - (Most) Complex building of node_modules is fully implemented as python application, because it requires a lot of control flow. - - Multiple outputs `["out" "lib"]` (explained below) -- Dedicated flattening of node_modules: - - Conflicts are resolved during flattening the node_modules folder in favor of the highest semver) - - Creates node_modules tree directly from package-lock.json informations. (Through the translator) - - Creates node_modules tree with other lock files (such as yarn-lock) most optimal. -- consume itself: - - lets you override/inject a package into your dream2nix project which is built with dream2nix. - -#### Usage - -in `projects.toml` set the `builder` attribute to `'strict-builder'` - -> ./projects.toml - -```toml -{{#include ../../../examples/nodejs_alternative_builder/projects.toml}} -``` - -#### Multiple outputs - -##### `lib` - package - -- consumable as bare package -- containing all files from the `source` ( _install-scripts_ already executed ) - -```bash - $lib - /nix/store/...-pname-1.0.0-lib - β”œβ”€β”€ cli.js - β”œβ”€β”€ ... - └── package.json -``` - -_install-scripts_ declared in `package.json` run in the following order: - -> preinstall -> install -> postinstall -> -> isolated from other packages -> -> if the isolation during installScript is causing you problems, let us know. - -##### `out` - standard composition - -- consumable by most users -- `{pname}/bin/...` contains all executables of this package - -```bash - $out: - /nix/store/...-pname-1.0.0 - β”œβ”€β”€ bin - β”‚ └── cli -> ../lib/cli.js - └── lib - β”œβ”€β”€ cli.js - β”œβ”€β”€ ... - β”œβ”€β”€ package.json - └── node_modules -``` - -#### DevShell - passthru.devShell - -Dedicated devShell with node_modules decoupled from the package. - -##### The Problem - -As you change lines in your codebase the input of the package changes. -Which then could lead to a change in the `hash` of the `node_modules` even if the `packages` of your project didn't change. -This basically means your nix shell / direnv could reload the shell every time you type. - -##### The Solution - -Decoupled `node_modules` from the inputs, dependending only on the current dependencies of the project and not on the source itself. diff --git a/docs/src/subsystems/php.md b/docs/src/subsystems/php.md deleted file mode 100644 index bc4eaba76c..0000000000 --- a/docs/src/subsystems/php.md +++ /dev/null @@ -1,43 +0,0 @@ -# PHP subsystem - -> !!! PHP support is experimental. \ -> !!! You can track the progress in -> [nix-community/dream2nix#240](https://github.com/nix-community/dream2nix/issues/240). - -This section documents the PHP subsystem. - -## Example - -An example of building [composer](https://github.com/composer/composer) using dream2nix. - -```nix -{{#include ../../../examples/php_composer/flake.nix}} -``` - -## Translators - -### composer-lock (pure) - -Translates `composer.lock` into a dream2nix lockfile. - -### composer-json (impure) - -Resolves dependencies in `composer.json` using `composer` to generate a -`composer.lock` lockfile, then invokes the `composer-lock` translator to -generate a dream2nix lockfile. - -### packagist (impure) - -Downloads a package version from [Packagist](https://packagist.org/). -Then uses `composer-lock` if a `composer.lock` file is present, -or `composer-json` otherwise to translate the package. - -## Builders - -### granular (pure) (default) - -Builds all dependencies in isolation, moving upwards to the top-level package. - -### simple (pure) - -Builds the package including all its dependencies in a single derivation. diff --git a/docs/src/subsystems/python.md b/docs/src/subsystems/python.md deleted file mode 100644 index d75c207559..0000000000 --- a/docs/src/subsystems/python.md +++ /dev/null @@ -1,28 +0,0 @@ -# Python subsystem - -This section documents the Python subsystem. - -## Examples -flake.nix -```nix -{{#include ../../../examples/python_pip/flake.nix}} -``` -## Translators - -### pip (impure) - -Can translate all pip compatible python projects, including projects managed with poetry or other tool-chains which integrate via `pyproject.toml`. - -This translator simply executes `pip download` on the given source and observes which sources are downloaded by pip. -The downside of this approach is, that this translator cannot be used with a granular builder. It does not understand the exact relation between the dependencies, and therefore it only allows to build all dependencies in one large derivation. - -#### **pip** optional translator arguments -```nix -{{#include ../../../src/subsystems/python/translators/pip/args.nix}} -``` - -## Builders - -### simple-builder (pure) (default) - -Builds a package including all its dependencies in a single derivation. diff --git a/docs/src/subsystems/rust.md b/docs/src/subsystems/rust.md deleted file mode 100644 index ce861604e1..0000000000 --- a/docs/src/subsystems/rust.md +++ /dev/null @@ -1,190 +0,0 @@ -# Rust subsystem - -This section documents the Rust subsystem. - -## Translators - -### cargo-lock (pure) - -Translates a `Cargo.lock` file to a dream2nix lockfile. - -### cargo-toml (impure) - -Translates a `Cargo.toml` file to a dream2nix lockfile by generating a `Cargo.lock` file first and then using `cargo-lock` translator on the generated lockfile. - -## Builders - -### build-rust-package (pure) - -Builds a package using `buildRustPackage` from `nixpkgs`. - -### crane (ifd) (default) - -Builds a package using [`crane`](https://github.com/ipetkov/crane). -This builder builds two separate derivations, one for dependencies and the other for your crate. -The dependencies derivation will be named `-deps` where `` is the name of the crate you are building. - -#### Setting profile and Cargo flags - -This can be done via setting environment variables: - -- `cargoTestFlags` and `cargoBuildFlags` are passed to `cargo` invocations for `checkPhase` and `buildPhase` respectively. -- `cargoTestProfile` and `cargoBuildProfile` are used as profiles while compiling for `checkPhase` and `buildPhase` respectively. - -#### Override gotchas - -This builder builds two separate derivations, one for your crate's dependencies and another for your crate. -This means that if you want to override stuff for the dependencies you need to use the `-deps` key for your override where `` is the name of the crate you are building. - -```nix -{ - # ... - packageOverrides = { - # this will apply to your crate - crate.my-overrides = { /* ... */ }; - # this will apply to your crate's dependencies - crate-deps.my-overrides = { /* ... */ }; - }; - # ... -} -``` - -#### On the IFD marking - -The `crane` builder utilizes IFD to clean the source your crates reside in. -This is needed to not rebuild the dependency only derivation everytime the source for your crates is changed. - -However this does not mean that the IFD will always be triggered. -If you are passing dream2nix a path source or a flake source, then IFD won't be triggered as these sources are already realized. -But if you are passing the result of a `pkgs.fetchFromGitHub` for example, this will trigger IFD since it is not already realized. - -### Specifying the Rust toolchain - -Specify an override for all packages that override the Rust toolchain used. -This can be done like so: - -```nix -{ - # ... - packageOverrides = { - # ... - "^.*".set-toolchain.overrideRustToolchain = old: { - inherit (pkgs) cargo rustc; - }; - # ... - }; - # ... -} -``` - -You can also of course override the toolchain for only certain crates: - -```nix -{ - # ... - packageOverrides = { - # ... - crate-name.set-toolchain.overrideRustToolchain = old: { - inherit (pkgs) cargo rustc; - }; - # ... - }; - # ... -} -``` - -#### `crane` notes - -If cross-compiling, keep in mind that it also takes `cargo` packages like so: - -```nix -{ - cargoHostTarget = cargo-package; - cargoBuildBuild = other-cargo-package; -} -``` - -where `cargoHostTarget` has the same meaning as coming from a `pkgsHostTarget`. -And `cargoBuildBuild` has the same meaning as coming from a `pkgsBuildBuild`. - -To override the toolchain for a specific package, you will need to set an override for both the dependencies and the main package derivation: - -```nix -let - toolchainOverride = old: { /* ... */ }; -in -{ - # ... - packageOverrides = { - # ... - crate-name.set-toolchain.overrideRustToolchain = toolchainOverride; - crate-name-deps.set-toolchain.overrideRustToolchain = toolchainOverride; - # ... - }; - # ... -} -``` - -#### Examples - -- Usage with [fenix](https://github.com/nix-community/fenix): -```nix -let - # ... - # we use the full toolchain derivation here as using - # only the cargo / rustc derivation *does not* work. - toolchain = fenix.packages.${system}.minimal.toolchain; - # ... -in -{ - # ... - packageOverrides = { - # for crane builder - "^.*".set-toolchain.overrideRustToolchain = old: {cargo = toolchain;}; - # for build-rust-package builder - "^.*".set-toolchain.overrideRustToolchain = old: { - cargo = toolchain; - rustc = toolchain; - }; - }; - # ... -} -``` - -- Usage with [oxalica's rust-overlay](https://github.com/oxalica/rust-overlay): -```nix -let - # ... - toolchain = rust-overlay.packages.${system}.rust; - # ... -in -{ - # ... - packageOverrides = { - # for crane builder - "^.*".set-toolchain.overrideRustToolchain = old: {cargo = toolchain}; - # for build-rust-package builder - "^.*".set-toolchain.overrideRustToolchain = old: { - cargo = toolchain; - rustc = toolchain; - }; - }; - # ... -} -``` - -### Specifying the `stdenv` - -`crane` supports specifying the `stdenv` like so: -```nix -{ - # ... - packageOverrides = { - # change all derivations' stdenv to clangStdenv - "^.*".set-stdenv.override = old: {stdenv = pkgs.clangStdenv;}; - }; - # ... -} -``` - -`build-rust-package` builder does not support specifying the `stdenv`.