Greenboot fails due to missing '/etc/ostree/remotes.d/fedora-iot.conf'

[pcdubs]

Describe the bug

Recent disk images no longer include the '/etc/ostree/remotes.d/fedora-iot.conf' file parsed by greenboot.

As a result when logging into a system:

Script '01_update_platforms_check.sh' FAILURE (exit code '1'). Continuing...
Boot Status is GREEN - Health Check SUCCESS

To Reproduce

Please describe the steps needed to reproduce the bug:

1. Boot recent Fedora 39 IoT disk image
2. SSH in

Expected behavior

No script failure.

Additional context

The system can still be updated, the remote repository information is now included in:

cat /sysroot/ostree/repo/config
[core]
repo_version=1
mode=bare

[remote "fedora-iot"]
url=https://ostree.fedoraproject.org/iot
gpgkeypath=/etc/pki/rpm-gpg/
gpg-verify=true
contenturl=mirrorlist=https://ostree.fedoraproject.org/iot/mirrorlist

[sysroot]
bootloader=none
readonly=true

[7flying]

• investigate why the change has happened first
• take a look at use ostree commands to fetch remote URLs greenboot#116

[miabbott]

Need a F40 IoT build that includes the RPM with ostree remote config to assess the current state of the problem.

Likely have to sort out why the remote config is landing under /sysroot/ostree/repo; possibly an rpm-ostree/ostree behavior?

[pcdubs]

In Fedora 40, there is no longer a greenboot failure as the remote is included in /etc/ostree/remotes.d/fedora-iot.conf as expected (provided by fedora-iot-config). The remote is no longer included in /sysroot/ostree/repo/config so there is no conflict and updates work.

In Fedora 39, the disk image - Fedora-IoT-39-20240201.0.aarch64.raw.xz - the remote has also been removed from /sysroot/ostree/repo/config, resulting in an error:

rpm-ostree status 
State: idle
Deployments:
● fedora-iot:fedora/stable/aarch64/iot
       OstreeRemoteStatus: Remote "fedora-iot" not found
                  Version: 39.20240201.0 (2024-02-01T18:02:07Z)
                   Commit: 01b59135918d7b465a8dfaad2429c4331fad2a7f2c60e64050bd569f3f06ab22

To get updates working again:

ostree remote add --set=gpg-verify=true --set=gpgkeypath=/etc/pki/rpm-gpg/ --set=contenturl=mirrorlist=https://ostree.fedoraproject.org/iot/mirrorlist  fedora-iot 'https://ostree.fedoraproject.org/iot'

[pcdubs]

This change seems to only affect new disk images, testing our GA image from 39.20231103.1 and upgraded was ok:

[root@localhost ~]# rpm-ostree status
State: idle
Deployments:
● fedora-iot:fedora/stable/aarch64/iot
                  Version: 39.20240201.0 (2024-02-01T18:02:07Z)
                   Commit: 01b59135918d7b465a8dfaad2429c4331fad2a7f2c60e64050bd569f3f06ab22
             GPGSignature: Valid signature by E8F23996F23218640CB44CBE75CF5AC418B8E74C

  fedora-iot:fedora/stable/aarch64/iot
                  Version: 39.20231103.1 (2023-11-03T18:17:43Z)
                   Commit: cc8d419f72d84ac24d0a95e235c0cdf72844f73d9d6f42a41fcddf23dfb34f7d
             GPGSignature: Valid signature by 6A51BBABBA3D5467B6171221809A8D7CEB10B464
[root@localhost ~]# cat /sysroot/ostree/repo/config 
[core]
repo_version=1
mode=bare

[remote "fedora-iot"]
url=https://ostree.fedoraproject.org/iot
gpgkeypath=/etc/pki/rpm-gpg/
gpg-verify=true
contenturl=mirrorlist=https://ostree.fedoraproject.org/iot/mirrorlist

[sysroot]
bootloader=none
readonly=true

Disk images from Fedora 38 included the remote in /etc/ostree/remotes.d/fedora-iot.conf and continue to work on upgrade.

[pcdubs]

• Disk image from Fedora-IoT-39.20231208.0-20231208.0.aarch64.raw includes the remote in /sysroot
• As of Fedora-IoT-39.20231214.0-20231214.0.aarch64.raw, the remote is no longer included at all.

● fedora-iot:fedora/stable/aarch64/iot
       OstreeRemoteStatus: Remote "fedora-iot" not found
                  Version: 39.20231214.0 (2023-12-15T01:47:31Z)
                   Commit: 536c86a9db2fa7b332a33f7d027ebf81226f52079990117563e2e37ba5f0259e

• ostree version didn't change ostree-2023.7-2.fc39.aarch64

• osbuild/images had a big jump from v0.15.0 > v0.18.0

We will need to add fedora-iot-config into F39 to fix this. Testing how best to do that as it will cause the legacy installer issues with duplicated remotes.

[achilleas-k]

Some info that might help track this down (adding to Paul's comment above).

osbuild/images v0.18.0 included the change to the packaged remote config. Instead of hard-coding the remote configuration in the disk image creation (which was replicating the disk creation kickstart), it was added in the fedora-iot-config package. This was done for both Fedora 39 and 40.
The change was merged into osbuild-composer here: osbuild/osbuild-composer@5b19bd6

This change was planned for a while with Peter and Antonio. The plan included packaging a remote configuration with fedora-iot-config, install the package in the iot-commit itself, and stop setting it during disk creation or in the Anaconda kickstart.

So if a disk image is built with newer versions of composer (versions that include osbuild/images v0.18.0+) but an older iot-commit is used (or one without the fedora-iot-config package), then there will be no remote configured in the image.

[pcdubs]

PR to add a script to detect and fix the remote

[pcdubs]

This is fixed in fedora-iot-config-1-1.fc39 and fedora-release-39-36 included in todays compose Fedora-IoT-39-20240226.0