DNF-JSON

[lavocatt]

DNF-JSON

Intro

The dnf-json tool serves as a gateway between DNF's python API and a client.
Dnf-json is a deamon. It is started by a systemd socket and answers http requests. These requests must contain a json respecting the format described below.
There are two kind of requests:

• depsolve
• dump

Depsolve request is used to solve the dependencies assuming an empty system.
Dump request is used to dump all the packages in each repos.

Description of the JSON interface to the service

Sample input for a depsolve request

{
  "command": "depsolve",
  "arguments": {
    "package-specs": [
      "kernel"
    ],
    "exclude-specs": null,
    "repos": [
      {
        "id": "0",
        "baseurl": "http://mirror.centos.org/centos/8-stream/extras/x86_64/os/",
        "ignoressl": false
      },
      {
        "id": "1",
        "baseurl": "http://mirror.centos.org/centos/8-stream/BaseOS/x86_64/os/",
        "ignoressl": false
      },
      {
        "id": "2",
        "baseurl": "http://mirror.centos.org/centos/8-stream/AppStream/x86_64/os/",
        "ignoressl": false
      }
    ],
    "cachedir": "/var/cache/osbuild-worker/rpmmd",
    "module_platform_id": "platform:el8",
    "arch": "x86_64"
  }
}

package-specs: the list of the packages to install
exclude-specs: the list of the packages to not instal
repos: the list of repos to load in the cache

Sample output

{
  "checksums": {
    "0": "sha256:3ba33fa21b2e6cd99f9ead37bf794f242b987b4a1b5fca6ca67584482f1d557f",
    "1": "sha256:0180a4fb50e2feec6d71c4648115d4d2d0c68d88f45d849f785778ca182d9f6e",
    "2": "sha256:6b376d8b475c31e5a51420472f4d34e3e2784972db537dc6e5bee0795de060c5"
  },
  "dependencies": [
    {
      "name": "xkeyboard-config",
      "epoch": 0,
      "version": "2.28",
      "release": "1.el8",
      "arch": "noarch",
      "repo_id": "2",
      "path": "Packages/xkeyboard-config-2.28-1.el8.noarch.rpm",
      "remote_location": "http://mirror.centos.org/centos/8-stream/AppStream/x86_64/os/Packages/xkeyboard-config-2.28-1.el8.noarch.rpm",
      "checksum": "sha256:a2aeabb3962859069a78acc288bc3bffb35485428e162caafec8134f5ce6ca67"
    },
    ...
  ]
}

Output truncated to show only one dependency.
checksums: the list of checksums of each in-cache repos
dependences: list of packages to install with their location, name and checksum

Interaction between the daemon side and the DNF's python API

for a depsolve request

classDiagram
    direction LR
    ForkingMixIn <|-- SystemDActivationSocketServer
    UnixStreamServer <|-- SystemDActivationSocketServer
    class SystemDActivationSocketServer{
        +server_bind()
        +BaseHTTPRequestHandler(BaseHTTPRequestHandler)
    }
    BaseHTTPRequestHandler <|-- DnfJsonRequestHandler
    class DnfJsonRequestHandler{
        +do_POST()
        -response_success()
        -response_failure()
    }
    SystemDActivationSocketServer o-- BaseHTTPRequestHandler
    DnfJsonRequestHandler *-- Solver
    class Solver{
        +Solver(repos: list, cache_dir: Path)
        +dump()
        +depsolve()
    }
    Solver *-- dnf_Base
    class dnf_Base{
        +conf
        +repos
        +resolve()
        +fill_sack()
    }

Loading

sequenceDiagram
    HttpRequest ->> DnfJsonRequestHandler: do_POST
    DnfJsonRequestHandler -->> DnfJsonRequestHandler: with process_lock
    DnfJsonRequestHandler ->> Solver: depsolve()
    Solver ->> dnf.Base: new()
    Solver ->> dnf.Base: set configuration
    loop for each reop
        Solver ->> dnf.Base: repos.add(repo)
    end
    Solver ->> dnf.Base: fill_sack(load_system_repo=False)
    Solver ->> dnf.Base: install_spec(package_list, exclude_list)
    Solver ->> dnf.Base: resolve()
    dnf.Base -->> Solver: list of transactions
    Solver -->> DnfJsonRequestHandler: answer list of packages in transactions
    DnfJsonRequestHandler -->> DnfJsonRequestHandler: write http response()
    DnfJsonRequestHandler -->> HttpRequest: request done

Loading

dnf configuration

        base.conf.fastestmirror = True
        base.conf.zchunk = False
        base.conf.module_platform_id = module_platform_id
        base.conf.config_file_path = "/dev/null"
        base.conf.persistdir = persistdir
        base.conf.cachedir = cachedir
        base.conf.substitutions['arch'] = arch
        base.conf.substitutions['basearch'] = dnf.rpm.basearch(arch)

Current procedure to depsolve dependencies for a custom image.

Double depsolve

To obtain the list of dependencies of a distribution we need to perform two solving request.
The first request contains the list of packages needed to create the base of the distributions minus all the weak dependencies.
We manually specify the list of the packages we don't want to have in the final distribution.

Followed by a second solving request for the customer's package list to install. This second request does not have weaks dependencies disabled, so it'll pull all the dependencies possible.

issues with this approach

The issue with the second depsolve, is that it will pull packages from the first solving request. So we will endup with a lot of duplicates.
And we can't really make only one request as some dependencies required by the second request might be masked by the excluded packages of the first one.

The ideal would be to trick DNF to think that the list of packages solved with the first request are installed on the system to perform the second request.

Hacks and Bodges

Cache cleaning

A lot of customer's customization are involving one-shot repositories that will be cached only once. Because of that we have an ever growing cache directory. We have to run a routine after each solving that cleans cache directories that aren't used since a long period of time.

Multiprocessing and the global lock

We have noticed that DNF's python API is leaking memory if we make create multiple dnf.Base objects one after the others. It might be due to out edgy way of using the API though. So to counteract that, each http request is handled in a separate process. This way the memory is automatically freed when the process is terminated.

[thozza]

Thanks for writing this down. My comments follow...

Current procedure to depsolve dependencies for a custom image.

Double depsolve

To obtain the list of dependencies of a distribution we need to perform two solving request. The first request contains the list of packages needed to create the base of the distributions minus all the weak dependencies. We manually specify the list of the packages we don't want to have in the final distribution.

... minus all the weak dependencies. - there is no such goal. The list of packages to exclude is specifically composed to achieve a desired base package set for the image. The excluded packages can be a weak dependency of some package, but also an optional package of an RPM package group, such as @core. We definitely do not exclude all weak dependencies, nor it is the goal. The weak dependencies settings used by dnf-json follows the OS / DNF default behavior.

I think that rather than talking about "dependencies of a distribution", we should talk about "package set of an image".

In addition, I would not say that we need to perform two solving requests, but it is what we ended up doing.

Followed by a second solving request for the customer's package list to install. This second request does not have weaks dependencies disabled, so it'll pull all the dependencies possible.

From use case point of view, I would say that we have two cases in composer:

1. Bootstrapping of an OS environment.

   • This means that we need to create an OS root with specific packages installed. In order to achieve it, we dep-solve the desired package set using DNF Python API as if it we wanted to install it on an empty OS (no packages installed). This results in all of the required dependencies being pulled in and we end up with a package set which can be considered a minimal OS environment based on the initial required package set.
   • This use case is required in multiple places in composer. Examples are:
     • buildroot environment used to run osbuild stages in.
     • base OS package set (ostree or not) without any Blueprint packages being installed.
     • anaconda ISO environment, which can boot and install an OS or a prebuilt payload.

2. Installation of additional user-requested packages in the image.

   • The user can customize the built image by requesting that additional packages be installed on the image, in addition to the default package set. These packages can be specifically excluded in the base image package set, but it still needs to be possible to install them. The process of dep-solving should not happen as if the user-requested package set was being installed on an empty OS (no packages installed), but rather installed on top of already existing OS (packages from the default image package set).

osbuild-composer currently uses approach (1.) for all package sets, while ideally it should use (2.) when handling Blueprint (user-provided) packages. Ideally, while composing the image OS package set (the actual payload), we should do (1.) + (2.) in one step and install the resulting combined package set.

issues with this approach

The issue with the second depsolve, is that it will pull packages from the first solving request. So we will endup with a lot of duplicates. And we can't really make only one request as some dependencies required by the second request might be masked by the excluded packages of the first one.

The important piece is that dep-solving the blueprint package set should not end up pulling in packages, which are not a transitive dependency of any of the requested packages, which would be already covered by the base image package set. The issue is that we currently see this undesired behavior, because we dep-solve blueprint packages as if there were no other packages installed in the image. This effectively ends up being a bootstrapping scenario, which it is not by definition.

To the point of ... we can't really make only one request ..., to be technically precise we can't combine the package specs into one and dep-solve them together in a single transaction. The main reason is that some of the explicitly excluded packages may be a hard-dependency of a user-requested package. It is impossible to determine that this situation will happen, without dep-solving the package set first. By that time it is already too late and we end up with a failed transaction.

The ideal would be to trick DNF to think that the list of packages solved with the first request are installed on the system to perform the second request.

Yes.

DNF seems to be able to read the already installed packages only from RPMDB, which is an unfortunate limitation for us.

However it turns out that by dep-solving two transactions in a row, gives what we need - osbuild/osbuild-composer#2125 (comment)