Ability to specify a location to a flatpak ref for a application?

[SpiderUnderUrBed]

Hello, I want to install sober declaratively using nix flatpak, however sober is not published on flathub or any repo, so I was wondering how i could specify the link to the ref file and have it automatically install? is this possible currently? if not then can it be worked on?
Here is the how they ship their flatpak ref:
https://sober.vinegarhq.org/sober.flatpakref

[gmodena]

Hey @SpiderUnderUrBed ,

I'm working on adding the capability to install from local flatpaks (as a way to improve testing).

I need to look into supporting generic URI, but that sounds like a nice feature to have.

[gmodena]

I need to look into supporting generic URI, but that sounds like a nice feature to have.

Supporting installation from a flatpakref URL is pretty straightforward.

What complicates things slightly is this behavior:
flatpak/flatpak#5460

flatpak fails to --reinstall or skip already installed apps from a flatpakref.
You can reproduce this issue by trying the following:

flatpak --user install https://sober.vinegarhq.org/sober.flatpakref
flatpak --user install --reinstall https://sober.vinegarhq.org/sober.flatpakref

The second command should throw an error. This would be nix-flatpak install/update lifecycle.

The only workaround I can think of is downloading the .flatpakref file and mapping the URL to the declared Name and RuntimeRepo.

[SpiderUnderUrBed]

What does this look like in nix? Like you specify:
https://sober.vinegarhq.org/sober.flatpakref
as a app, does nix flatpak even support that?

Also does this mean generic URI's cant be supported? A way of implementation would require a potentially different install/update lifecycle for file paths. Or URLS to flatpakrefs.

[Bear-03]

This is my current workaround for this issue.

I noticed that .flatpakref and .flatpakrepo files have very similar keys, the only significant thing that changes is the header itself, so I just added the ref file as an input to my flake, and then I replace the header before feeding it to nix-flatpak so it can read it as a .flatpakrepo.

Seems to work flawlessly so far.

[gmodena]

The only workaround I can think of is downloading the .flatpakref file and mapping the URL to the declared Name and RuntimeRepo.

I did some further testing. This might require some adjustments to the state management logic to avoid breaking configurations that declare services.flatpak.uninstallUnmanaged = true.

When you run flatpak install --from <flatpakref>, a new repository is added to Flatpak's configuration.

For example, when I install with

flatpak --user install --from https://sober.vinegarhq.org/sober.flatpakref
I'm promted with:

Should the remote be kept for future installations? [Y/n]

Even If I select n, flatpak will still add a remote:

$ flatpak remotes
Name         Options
flathub      system
flathub-beta system
flathub      user
sober-origin user,no-enumerate

Unless the repository is explicitly declared in services.flatpak.remotes, it will be deleted during the next activation, and the application will also be deleted. Howevert, unless explicitly removed from the packages declaration, the application would still be re-installed as part of the activation. That's a rather wasteful operation IMHO (the application would need to be re-downloaded each time an activation is triggered).

@Bear-03 gave a neat example of how to automate remote declaration configs.

A flatpakref will contain all the info you need to declare your remote and application manually. Let's take a look at Sober:

$ curl -s "https://sober.vinegarhq.org/sober.flatpakref"
[Flatpak Ref]
Title=Sober
Name=org.vinegarhq.Sober
Url=https://sober.vinegarhq.org/repo/
SuggestRemoteName=sober
Homepage=https://sober.vinegarhq.org
Icon=https://sober.vinegarhq.org/sober.svg
RuntimeRepo=https://dl.flathub.org/repo/flathub.flatpakrepo
IsRuntime=false
GPGKey=<redacted>

With this, you can declare a remote and package as:

services.flatpak.remotes = lib.mkOptionDefault [{
    name = "sober"; # from SuggestRemoteName
    location = "https://sober.vinegarhq.org/repo/"; # from Url
  }];

services.flatpak.packages = [
   { appId = "org.vinegarhq.Sober"; origin = "sober"; }
];

Where appId comes from the flatpakref Name field and origin is the name of the remote we added.

You could follow the example @Bear-03 provided and automate parsing of the flapakref.

All in all, I think this approach is less error prone than trying to manage remote /application state implicitly.

Would it be useful if nix-flatpak provided helper functions to work with flatpakrepo / flatpakref files?

[Bear-03]

With this, you can declare a remote and package as:

services.flatpak.remotes = lib.mkOptionDefault [{
    name = "sober"; # from SuggestRemoteName
    location = "https://sober.vinegarhq.org/repo/"; # from Url
  }];

services.flatpak.packages = [
     { appId = "org.vinegarhq.Sober"; origin = "sober"; }
];

I did try this before and unfortunately, in my case it causes the following error:

oct 06 23:27:01 bear-laptop systemd[1]: Starting flatpak-managed-install.service...
oct 06 23:27:02 bear-laptop flatpak[46214]: system: Added remote sober to https://sober.vinegarhq.org/repo/
oct 06 23:27:02 bear-laptop 7yzcg37k8xn5lvryngwdrpjzyl1m4a12-flatpak-managed-install[46214]: error: Signature made dom 15 sep 2024 18:55:44 using RSA key ID DAF8BEF8138D4B65
oct 06 23:27:02 bear-laptop 7yzcg37k8xn5lvryngwdrpjzyl1m4a12-flatpak-managed-install[46214]: Can't check signature: public key not found
oct 06 23:27:02 bear-laptop systemd[1]: flatpak-managed-install.service: Main process exited, code=exited, status=1/FAILURE

As I understand it, it's missing the GPG key tht is included in the .flatpakref. That's why my next approach was to just keep the file and patch the header.

I don't know how nix-flatpak works internally, but wouldn't it be possible to know that a certain repo does not have to be uninstalled because it is (implicitly) needed for a flatpakref package? If the package list is traversed and some packages are flatpakrefs, the names of their repos should be extracted from the files and even if uninstallUnmanaged = true, nix-flatpak would be able to use those names to ignore and keep the repos, just like with those in the remotes list.

[SpiderUnderUrBed]

Also to answer your proposal, nix-flatpak should definitely add helper functions, I tried Bear-03's configuration and it yielded no errors, so I assume its done its job (I didn't re-install sober but I will try later). However, the current method is quite unintuitive and probably with caveats.

[gmodena]

Hey @Bear-03

As I understand it, it's missing the GPG key tht is included in the .flatpakref. That's why my next approach was to just keep the file and patch the header.

Good call. I think the pub key got cached in my experiments. I tried on a freshly built vm, and can repro that error.

I don't know how nix-flatpak works internally, but wouldn't it be possible to know that a certain repo does not have to be uninstalled because it is (implicitly) needed for a flatpakref package? I

Yep, it can be done pretty much the way you describe it. I have a PoC that needs a little flashing out (mostly internal APIs) and some more testing.

I do like your approach though, because it makes remote declaration explicit.

I was about to suggest you could pass the GPG key by injecting custom args = "..." in the remote declaration, but I am afraid that option is broken right now (it's something I've kept around for dev/tests).

[Bear-03]

I was about to suggest you could pass the GPG key by injecting custom args = "..." in the remote declaration, but I am afraid that option is broken right now (it's something I've kept around for dev/tests).

Even then, in the long run I think it would be much cleaner to allow passing .flatpakrefs to nix-flatpak directly (either via the packages option or a completely new one) instead of having to manually extract the data from within them, which is harder to maintain and more error prone.

[SpiderUnderUrBed]

Also the ability to specify .flatpakrepo's could come in handy (in the case of local flatpak development or there might be a case where you don't have a repo at all). Specifying a .flatpakrepo should also be a feature at some point.

[Bear-03]

Isn't that already possible though? you can provide a local path with file://<absolute path to .flatpakrepo> like I did in my config. Although now that I think of it, the protocol could be implicit if a path is provided instead of a url...

[gmodena]

Isn't that already possible though? you can provide a local path with file://<absolute path to .flatpakrepo> like I did in my config. Although now that I think of it, the protocol could be implicit if a path is provided instead of a url...

Specifying a protocol is, in my opinion, good practice, but in case of remotes configs it is not strictly required. The location attribute is passed to flatpak remote-add, and Flatpak will resolve a local path even if the protocol is not specified.

[gmodena]

@Bear-03 @SpiderUnderUrBed I still need to add some unit tests before resolving, but if you want to give it a go #80 should be in a decent enough shape.

You can test the feature by tracking the install-flatpakref branch in your config:

...
nix-flatpak.url = "github:gmodena/nix-flatpak/install-flatpakref";
...

[SpiderUnderUrBed]

I am getting an issue with the whole install/reinstall cycle as you said:

Oct 11 00:48:44 daspidercave systemd[1]: Starting flatpak-managed-install.service...
Oct 11 00:48:48 daspidercave sabklji2jdalpb044z360f89k2cdlrmm-flatpak-managed-install[48600]: The following refs are installed from remote 'sober':
Oct 11 00:48:48 daspidercave sabklji2jdalpb044z360f89k2cdlrmm-flatpak-managed-install[48600]:    1) app/org.vinegarhq.Sober/x86_64/master
Oct 11 00:48:48 daspidercave sabklji2jdalpb044z360f89k2cdlrmm-flatpak-managed-install[48600]: Remove them? [y/n]: n
Oct 11 00:48:48 daspidercave sabklji2jdalpb044z360f89k2cdlrmm-flatpak-managed-install[48600]: error: Can't remove remote 'sober' with installed refs
Oct 11 00:48:48 daspidercave systemd[1]: flatpak-managed-install.service: Main process exited, code=exited, status=1/FAILURE
Oct 11 00:48:48 daspidercave systemd[1]: flatpak-managed-install.service: Failed with result 'exit-code'.
Oct 11 00:48:48 daspidercave systemd[1]: Failed to start flatpak-managed-install.service.
Oct 11 00:48:48 daspidercave systemd[1]: flatpak-managed-install.service: Consumed 373ms CPU time, 16.9M memory peak, 44.7K incoming IP traffic, 9.6K outgoing IP traffic.

Here is my nixos config files (under flatpak.nix, also forgive me if it looks a little weird, I haven't cleaned up the comments and had a rebase mishap):
https://github.com/SpiderUnderUrBed/nixos-files

[gmodena]

@SpiderUnderUrBed thanks for giving it a go!

Could you paste me the content of $HOME/.local/state/home-manager/gcroots/flatpak-state.json (if you load nix-flatpak as an HM module) or /nix/var/nix/gcroots/flatpak-state.json (if you load nix-flatpak as a nixos module)?

[SpiderUnderUrBed]

I load nix-flatpak as a nixos module:
[spiderunderurbed@daspidercave:~]$ cat /nix/var/nix/gcroots/flatpak-state.json
{"overrides":{},"packages":["moe.launcher.the-honkers-railway-launcher","moe.launcher.an-anime-game-launcher","org.vinegarhq.Sober","io.gitlab.idevecore.Pomodoro","org.inkscape.Inkscape","org.signal.Signal","org.kde.knights","com.opera.Opera","org.nickvision.tubeconverter","io.kapsa.drive","com.rawtherapee.RawTherapee","org.eclipse.Java","org.rncbc.qpwgraph","com.google.Chrome","net.lutris.Lutris","com.valvesoftware.Steam","org.gnome.Notes","io.github.seadve.Mousai","org.torproject.torbrowser-launcher","org.kde.kdenlive","im.fluffychat.Fluffychat","org.kde.krita","com.system76.Popsicle","xyz.xclicker.xclicker","com.github.tchx84.Flatseal","com.obsproject.Studio","org.gimp.GIMP","com.visualstudio.code","org.qbittorrent.qBittorrent","org.nickvision.tubeconverter","de.haeckerfelix.Shortwave"],"remotes":["flathub","gol","flathub-beta","flathub","sober"]}

[gmodena]

@SpiderUnderUrBed thanks for sharing. flatpak-state.json look ok to me. Unfortunately I have not been able to repro your error:

Oct 11 00:48:48 daspidercave sabklji2jdalpb044z360f89k2cdlrmm-flatpak-managed-install[48600]: The following refs are installed from remote 'sober':
Oct 11 00:48:48 daspidercave sabklji2jdalpb044z360f89k2cdlrmm-flatpak-managed-install[48600]:    1) app/org.vinegarhq.Sober/x86_64/master
Oct 11 00:48:48 daspidercave sabklji2jdalpb044z360f89k2cdlrmm-flatpak-managed-install[48600]: Remove them? [y/n]: n
Oct 11 00:48:48 daspidercave sabklji2jdalpb044z360f89k2cdlrmm-flatpak-managed-install[48600]: error: Can't remove remote 'sober' with installed refs

the remote should have been removed only after the package was uninstall. Could you look in your system logs for failures wrt uninstalling org.vinegarhq.Sober ?

Could you try to uninstall the package and remote manually with:

flatpak uninstall org.vinegarhq.Sober
flatpak remote-delete sober

[SpiderUnderUrBed]

This error should be replicated if you install a remote, like "sober" which is a remote automatically made when you install from a flatpak ref. Then you install something from that remote, which is sober itself (from the flatpakref). I am confident this happens because uninstallUnmanaged is true, I don't know if you want this behavior to exist though, should it just skip uninstalling remotes with an existing ref (flatpakref installed from it)? or should it try to uninstall the ref first then the remote?

[gmodena]

This error should be replicated if you install a remote, like "sober" which is a remote automatically made when you install from a flatpak ref.

I did try to repro with uninstallUnmanaged=true, and a remote pointing to soberPatchedPath (same string replacement logic you linked).

First activation looks like this:

services.flatpak.uninstallUnmanaged = true;
services.flatpak.remotes = [
           {
               name = "flathub";
               location = "https://dl.flathub.org/repo/flathub.flatpakrepo";
           }
           {
               name = "sober";
               location = "file://${soberPatchedPath}";
           }
 ];
services.flatpak.packages = [
   { appId = "org.vinegarhq.Sober"; origin = "sober"; }
];

Remote and application have been installed.

Second activation:

services.flatpak.uninstallUnmanaged = true;
services.flatpak.remotes = [
            {
                name = "flathub";
                location = "https://dl.flathub.org/repo/flathub.flatpakrepo";
            }
  ];

services.flatpak.packages = [];

Remote and application are not installed in this generation (they have both been uninstalled on activation).

should it just skip uninstalling remotes with an existing ref (flatpakref installed from it)? or should it try to uninstall the ref first then the remote?

The expected behavior is that the flatpak-managed-install script will first attempt to uninstall the application and then remove the remote. If uninstalling the application fails, I would expect the script to abort.

I don't know why your system ended up in a state where the remote is uninstallable because of a dangling ref.

Could it be that at some point you installed the remote/app both declaratively and manually with flatpak install ...? Do you see multiple references to sober origins in flatpak remotes -v ? Similarly, do you see multiple Sober instances in flatpak list --app -v?

Maybe a failsafe is needed for this scenario:

Oct 11 00:48:48 daspidercave sabklji2jdalpb044z360f89k2cdlrmm-flatpak-managed-install[48600]: The following refs are installed from remote 'sober':
Oct 11 00:48:48 daspidercave sabklji2jdalpb044z360f89k2cdlrmm-flatpak-managed-install[48600]:    1) app/org.vinegarhq.Sober/x86_64/master
Oct 11 00:48:48 daspidercave sabklji2jdalpb044z360f89k2cdlrmm-flatpak-managed-install[48600]: Remove them? [y/n]: n

Defaulting to y when flatpak ask to remove the refs should clean up state.

[SpiderUnderUrBed]

[spiderunderurbed@daspidercave:~]$ flatpak remotes -v
F: Opening system flatpak installation at path /var/lib/flatpak
F: Opening user flatpak installation at path /home/spiderunderurbed/.local/share/flatpak
flathub system
flathub-beta    system
gol     system
sober   system
flathub user
sober   user

https://pastebin.com/HBDFHbYv
The actual issue you eluded too is present:

Remote ‘sober’ found in multiple installations:

   1) system
   2) user

and I already sent a link to my repo, I think I found the source of the issue, its like you said, multiple remotes. I don't know if you tested dangling refs/my issue but I think you should be able to get the same error if you reproduce it that way. Then you can address this bug with your suggested failsafe.

[gmodena]

Thanks for confirming.

Then you can address this bug with your suggested failsafe.

I'll open a dedicate PR for this.