Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

systemd-sysusers fails to create users needed for nm-openvpn #89

Open
RoyalOughtness opened this issue Nov 29, 2024 · 8 comments
Open

systemd-sysusers fails to create users needed for nm-openvpn #89

RoyalOughtness opened this issue Nov 29, 2024 · 8 comments
Assignees
@rehashedsalt
Labels
bug Something isn't working

Comments

@RoyalOughtness
Copy link
Collaborator

No description provided.

@RoyalOughtness RoyalOughtness added the bug Something isn't working label Nov 29, 2024
@eXsoR65
Copy link

eXsoR65 commented Nov 30, 2024 •
edited
Loading

I wanted to clearly that the issue is not just seen for nm-openvpn user, but most users/groups being created by systemd-sysusers.service.

Steps I'm taking and were I'm seen the issue.
(note: I'm doing this on physical hardware Lenovo X1 Carbon, though not sure that matters at all.)

  1. Install Fedora Silverblue 41, checked logs and systemd-sysusers are working correctly.
  2. Re-based to my spin of Wayblue: https://github.com/eXsoR65/Silverwing following as instructed to rebasing twice (unsigned image and then the singed image).
  3. Once in the Signed image, I tried to open Virt-Manager but it give me an error. This is when I had forgotten I needed to add me user to libvert group to have permissions needed. When trying to do that it status the group doesn't exists. which is what lead me to finding that systemd-sysusers if failing to create the needed users and groups for libvirt and virt-manager to work.

Edit: Adding logs and more context.

Logs from journalctl --unit systemd-sysusers

You can see that from boot it tries a few times but always exits and fails to crate all the users and groups.

-- Boot 13f69774849546489c33ba34fb1a99f6 --
Nov 29 21:10:32 fedora systemd[1]: systemd-sysusers.service: Deactivated successfully.
Nov 29 21:10:32 fedora systemd[1]: Stopped systemd-sysusers.service - Create System Users.
Nov 29 21:10:33 fedora systemd[1]: Starting systemd-sysusers.service - Create System Users...
Nov 29 21:10:33 fedora systemd-sysusers[837]: Creating group 'clevis' with GID 970.
Nov 29 21:10:33 fedora systemd-sysusers[837]: Creating group 'dhcpcd' with GID 969.
Nov 29 21:10:33 fedora systemd-sysusers[837]: Creating group 'gluster' with GID 968.
Nov 29 21:10:33 fedora systemd-sysusers[837]: Creating group 'libvirt' with GID 967.
Nov 29 21:10:33 fedora systemd-sysusers[837]: Creating group 'nm-openconnect' with GID 966.
Nov 29 21:10:33 fedora systemd-sysusers[837]: Creating group 'nm-openvpn' with GID 965.
Nov 29 21:10:33 fedora systemd-sysusers[837]: Creating group 'openvpn' with GID 964.
Nov 29 21:10:33 fedora systemd-sysusers[837]: Creating group 'qemu' with GID 107.
Nov 29 21:10:33 fedora systemd-sysusers[837]: Creating group 'sddm' with GID 963.
Nov 29 21:10:33 fedora systemd-sysusers[837]: Creating group 'wsdd' with GID 962.
Nov 29 21:10:33 fedora systemd-sysusers[837]: Creating group 'adbusers' with GID 961.
Nov 29 21:10:33 fedora systemd-sysusers[837]: Creating user 'clevis' (Clevis Decryption Framework unprivileged user) with >
Nov 29 21:10:33 fedora systemd-sysusers[837]: Creating user 'dhcpcd' (Minimalistic DHCP client) with UID 969 and GID 969.
Nov 29 21:10:33 fedora systemd-sysusers[837]: Creating user 'gluster' (GlusterFS daemons) with UID 968 and GID 968.
Nov 29 21:10:33 fedora systemd-sysusers[837]: Creating user 'nm-openconnect' (NetworkManager user for OpenConnect) with UI>
Nov 29 21:10:33 fedora systemd-sysusers[837]: Creating user 'nm-openvpn' (Default user for running openvpn spawned by Netw>
Nov 29 21:10:33 fedora systemd-sysusers[837]: Creating user 'openvpn' (OpenVPN) with UID 964 and GID 964.
Nov 29 21:10:33 fedora systemd-sysusers[837]: Creating user 'qemu' (qemu user) with UID 107 and GID 107.
Nov 29 21:10:33 fedora systemd-sysusers[837]: Creating user 'sddm' (SDDM Greeter Account) with UID 963 and GID 963.
Nov 29 21:10:33 fedora systemd-sysusers[837]: Creating user 'wsdd' (Web Services Dynamic Discovery host daemon) with UID 9>
Nov 29 21:10:33 fedora systemd-sysusers[837]: /etc/gshadow: Group "nm-openconnect" already exists.
Nov 29 21:10:33 fedora systemd[1]: systemd-sysusers.service: Main process exited, code=exited, status=1/FAILURE
Nov 29 21:10:33 fedora systemd[1]: systemd-sysusers.service: Failed with result 'exit-code'.
Nov 29 21:10:33 fedora systemd[1]: Failed to start systemd-sysusers.service - Create System Users.
Nov 29 21:10:34 fedora systemd[1]: Starting systemd-sysusers.service - Create System Users...
Nov 29 21:10:34 fedora systemd-sysusers[1032]: Creating group 'clevis' with GID 970.
Nov 29 21:10:34 fedora systemd-sysusers[1032]: Creating group 'dhcpcd' with GID 969.
Nov 29 21:10:34 fedora systemd-sysusers[1032]: Creating group 'gluster' with GID 968.
Nov 29 21:10:34 fedora systemd-sysusers[1032]: Creating group 'libvirt' with GID 967.
Nov 29 21:10:34 fedora systemd-sysusers[1032]: Creating group 'nm-openconnect' with GID 966.
Nov 29 21:10:34 fedora systemd-sysusers[1032]: Creating group 'nm-openvpn' with GID 965.
Nov 29 21:10:34 fedora systemd-sysusers[1032]: Creating group 'openvpn' with GID 964.
Nov 29 21:10:34 fedora systemd-sysusers[1032]: Creating group 'qemu' with GID 107.
Nov 29 21:10:34 fedora systemd-sysusers[1032]: Creating group 'sddm' with GID 963.
Nov 29 21:10:34 fedora systemd-sysusers[1032]: Creating group 'wsdd' with GID 962.
Nov 29 21:10:34 fedora systemd-sysusers[1032]: Creating group 'adbusers' with GID 961.
Nov 29 21:10:34 fedora systemd-sysusers[1032]: Creating user 'clevis' (Clevis Decryption Framework unprivileged user) with>
Nov 29 21:10:34 fedora systemd-sysusers[1032]: Creating user 'dhcpcd' (Minimalistic DHCP client) with UID 969 and GID 969.
Nov 29 21:10:34 fedora systemd-sysusers[1032]: Creating user 'gluster' (GlusterFS daemons) with UID 968 and GID 968.
Nov 29 21:10:34 fedora systemd-sysusers[1032]: Creating user 'nm-openconnect' (NetworkManager user for OpenConnect) with U>
Nov 29 21:10:34 fedora systemd-sysusers[1032]: Creating user 'nm-openvpn' (Default user for running openvpn spawned by Net>
Nov 29 21:10:34 fedora systemd-sysusers[1032]: Creating user 'openvpn' (OpenVPN) with UID 964 and GID 964.
Nov 29 21:10:34 fedora systemd-sysusers[1032]: Creating user 'qemu' (qemu user) with UID 107 and GID 107.
Nov 29 21:10:34 fedora systemd-sysusers[1032]: Creating user 'sddm' (SDDM Greeter Account) with UID 963 and GID 963.
Nov 29 21:10:34 fedora systemd-sysusers[1032]: Creating user 'wsdd' (Web Services Dynamic Discovery host daemon) with UID >
Nov 29 21:10:34 fedora systemd-sysusers[1032]: /etc/gshadow: Group "nm-openconnect" already exists.
Nov 29 21:10:34 fedora systemd[1]: systemd-sysusers.service: Main process exited, code=exited, status=1/FAILURE
Nov 29 21:10:34 fedora systemd[1]: systemd-sysusers.service: Failed with result 'exit-code'.
Nov 29 21:10:34 fedora systemd[1]: Failed to start systemd-sysusers.service - Create System Users.

Running a cat /etc/group you can see that all groups that systemd-sysusers tried to crate where not created.

root:x:0:
wheel:x:10:exsor
exsor:x:1000:
sddm:x:995:

Running a cat /etc/passwd you can see that all users that systemd-sysusers treod to create where not created.

root:x:0:0:root:/root:/bin/bash
exsor:x:1000:1000:eXsoR:/var/home/exsor:/bin/bash
sddm:x:992:995:SDDM Greeter Account:/var/lib/sddm:/usr/sbin/nologin

@eXsoR65
Copy link

eXsoR65 commented Nov 30, 2024

I forgot to add if I cat /etc/gshadow you can see there a ton so f groups and users that are not in /etc/{passwd}{group}.

root:::
bin:::
daemon:::
sys:::
adm:::
tty:::
disk:::
lp:::
mem:::
kmem:::
wheel:::exsor
cdrom:::
mail:::
man:::
dialout:::
floppy:::
games:::
tape:::
video:::
ftp:::
lock:::
audio:::
nobody:::
users:::
utmp:::
utempter:::
ssh_keys:::
systemd-journal:::
dbus:::
polkitd:::
etcd:::
dip:::
cgred:::
tss:::
avahi-autoipd:::
rpc:::
sssd:::
dockerroot:::
rpcuser:::
nfsnobody:::
kube:::
sshd:::
chrony:::
tcpdump:::
input:::
systemd-timesync:::
systemd-network:::
systemd-resolve:::
systemd-bus-proxy:::
cockpit-ws:::
apache:!::
avahi:!::
geoclue:!::
usbmuxd:!::
printadmin:!::
brlapi:!::
rtkit:!::
pipewire:!::
unbound:!::
nm-openconnect:!::
wsdd:!::
openvpn:!::
nm-openvpn:!::
flatpak:!::
colord:!::
gdm:!::
gnome-initial-setup:!::
vboxsf:!::
dnsmasq:!::
power:!::
plocate:!::
kvm:!*::
render:!*::
sgx:!*::
gamemode:!*::
gnome-remote-desktop:!*::
passim:!*::
systemd-coredump:!*::
systemd-oom:!*::
exsor:!::
sddm:!::

@eXsoR65
Copy link

eXsoR65 commented Dec 1, 2024 •
edited
Loading

@rehashedsalt I have confirmed the issue is cause by the groups:

  • nm-openconnect
  • wsdd
  • openvpn
  • nm-openvpn

Once I removed them with grpck I then ran systemd-sysusers and it was able to run and create all the users and groups.

I'll provide logs in a bit.

@sgallag-insta
Copy link
Collaborator

Interestingly my experience is a bit different. I've been using the wayblue hyprland nvidia-open image for a while.

My systemd-sysusers logs, with no adjustments:

-- Boot e2853bc1259f48c98f806b8ee48fa7a5 --
Nov 30 12:20:45 systemd[1]: systemd-sysusers.service: Deactivated successfully.
Nov 30 12:20:45 systemd[1]: Stopped systemd-sysusers.service - Create System Users.
Nov 30 12:20:46 systemd[1]: Starting systemd-sysusers.service - Create System Users...
Nov 30 12:20:46 systemd[1]: Finished systemd-sysusers.service - Create System Users.

However, I too have been having problems with groups after wayblue upgraded to Fedora 41. Relevant error logs:

Dec 01 09:20:17 systemd-tmpfiles[385]: /usr/lib/tmpfiles.d/static-nodes-permissions.conf:12: Failed to resolve group 'audio': No such process
Dec 01 09:20:17 systemd-tmpfiles[385]: /usr/lib/tmpfiles.d/static-nodes-permissions.conf:13: Failed to resolve group 'audio': No such process
Dec 01 09:20:17 systemd-tmpfiles[385]: /usr/lib/tmpfiles.d/static-nodes-permissions.conf:14: Failed to resolve group 'disk': No such process
Dec 01 09:20:17 systemd-tmpfiles[385]: /usr/lib/tmpfiles.d/static-nodes-permissions.conf:18: Failed to resolve group 'kvm': No such process
Dec 01 09:20:17 systemd-tmpfiles[385]: /usr/lib/tmpfiles.d/static-nodes-permissions.conf:19: Failed to resolve group 'kvm': No such process
Dec 01 09:20:17 systemd-tmpfiles[385]: /usr/lib/tmpfiles.d/static-nodes-permissions.conf:20: Failed to resolve group 'kvm': No such process
Dec 01 09:20:17 systemd-tmpfiles[428]: /usr/lib/tmpfiles.d/systemd.conf:11: Failed to resolve group 'utmp': No such process
Dec 01 09:20:17 systemd-tmpfiles[428]: /usr/lib/tmpfiles.d/var.conf:15: Failed to resolve group 'utmp': No such process
Dec 01 09:20:17 systemd-tmpfiles[428]: /usr/lib/tmpfiles.d/var.conf:16: Failed to resolve group 'utmp': No such process
Dec 01 09:20:17 systemd-tmpfiles[428]: /usr/lib/tmpfiles.d/var.conf:17: Failed to resolve group 'utmp': No such process
Dec 01 09:20:20 kernel: 
Dec 01 09:20:22 systemd-udevd[572]: /usr/lib/udev/rules.d/50-udev-default.rules:38 Unknown group 'tty', ignoring.
Dec 01 09:20:22 systemd-udevd[572]: /usr/lib/udev/rules.d/50-udev-default.rules:39 Unknown group 'tty', ignoring.
Dec 01 09:20:22 systemd-udevd[572]: /usr/lib/udev/rules.d/50-udev-default.rules:40 Unknown group 'tty', ignoring.
Dec 01 09:20:22 systemd-udevd[572]: /usr/lib/udev/rules.d/50-udev-default.rules:41 Unknown group 'tty', ignoring.
Dec 01 09:20:22 systemd-udevd[572]: /usr/lib/udev/rules.d/50-udev-default.rules:44 Unknown group 'kmem', ignoring.
Dec 01 09:20:22 systemd-udevd[572]: /usr/lib/udev/rules.d/50-udev-default.rules:46 Unknown group 'input', ignoring.
Dec 01 09:20:22 systemd-udevd[572]: /usr/lib/udev/rules.d/50-udev-default.rules:49 Unknown group 'video', ignoring.
Dec 01 09:20:22 systemd-udevd[572]: /usr/lib/udev/rules.d/50-udev-default.rules:50 Unknown group 'video', ignoring.
Dec 01 09:20:22 systemd-udevd[572]: /usr/lib/udev/rules.d/50-udev-default.rules:51 Unknown group 'video', ignoring.
Dec 01 09:20:22 systemd-udevd[572]: /usr/lib/udev/rules.d/50-udev-default.rules:52 Unknown group 'video', ignoring.
Dec 01 09:20:22 systemd-udevd[572]: /usr/lib/udev/rules.d/50-udev-default.rules:53 Unknown group 'video', ignoring.
Dec 01 09:20:22 systemd-udevd[572]: /usr/lib/udev/rules.d/50-udev-default.rules:54 Unknown group 'video', ignoring.
Dec 01 09:20:22 systemd-udevd[572]: /usr/lib/udev/rules.d/50-udev-default.rules:56 Unknown group 'render', ignoring.
Dec 01 09:20:22 systemd-udevd[572]: /usr/lib/udev/rules.d/50-udev-default.rules:57 Unknown group 'render', ignoring.
Dec 01 09:20:22 systemd-udevd[572]: /usr/lib/udev/rules.d/50-udev-default.rules:58 Unknown group 'render', ignoring.
Dec 01 09:20:22 systemd-udevd[572]: /usr/lib/udev/rules.d/50-udev-default.rules:60 Unknown group 'sgx', ignoring.
Dec 01 09:20:22 systemd-udevd[572]: /usr/lib/udev/rules.d/50-udev-default.rules:61 Unknown group 'sgx', ignoring.
Dec 01 09:20:22 systemd-udevd[572]: /usr/lib/udev/rules.d/50-udev-default.rules:67 Unknown group 'audio', ignoring.
Dec 01 09:20:22 systemd-udevd[572]: /usr/lib/udev/rules.d/50-udev-default.rules:77 Unknown group 'audio', ignoring.
Dec 01 09:20:22 systemd-udevd[572]: /usr/lib/udev/rules.d/50-udev-default.rules:78 Unknown group 'audio', ignoring.
Dec 01 09:20:22 systemd-udevd[572]: /usr/lib/udev/rules.d/50-udev-default.rules:79 Unknown group 'video', ignoring.
Dec 01 09:20:22 systemd-udevd[572]: /usr/lib/udev/rules.d/50-udev-default.rules:81 Unknown group 'lp', ignoring.
Dec 01 09:20:22 systemd-udevd[572]: /usr/lib/udev/rules.d/50-udev-default.rules:82 Unknown group 'lp', ignoring.
Dec 01 09:20:22 systemd-udevd[572]: /usr/lib/udev/rules.d/50-udev-default.rules:83 Unknown group 'lp', ignoring.
Dec 01 09:20:22 systemd-udevd[572]: /usr/lib/udev/rules.d/50-udev-default.rules:84 Unknown group 'lp', ignoring.
Dec 01 09:20:22 systemd-udevd[572]: /usr/lib/udev/rules.d/50-udev-default.rules:85 Unknown group 'lp', ignoring.
Dec 01 09:20:22 systemd-udevd[572]: /usr/lib/udev/rules.d/50-udev-default.rules:86 Unknown group 'lp', ignoring.
Dec 01 09:20:22 systemd-udevd[572]: /usr/lib/udev/rules.d/50-udev-default.rules:88 Unknown group 'disk', ignoring.
Dec 01 09:20:22 systemd-udevd[572]: /usr/lib/udev/rules.d/50-udev-default.rules:96 Unknown group 'disk', ignoring.
Dec 01 09:20:22 systemd-udevd[572]: /usr/lib/udev/rules.d/50-udev-default.rules:97 Unknown group 'disk', ignoring.
Dec 01 09:20:22 systemd-udevd[572]: /usr/lib/udev/rules.d/50-udev-default.rules:98 Unknown group 'disk', ignoring.
Dec 01 09:20:22 systemd-udevd[572]: /usr/lib/udev/rules.d/50-udev-default.rules:99 Unknown group 'disk', ignoring.
Dec 01 09:20:22 systemd-udevd[572]: /usr/lib/udev/rules.d/50-udev-default.rules:100 Unknown group 'disk', ignoring.
Dec 01 09:20:22 systemd-udevd[572]: /usr/lib/udev/rules.d/50-udev-default.rules:101 Unknown group 'disk', ignoring.
Dec 01 09:20:22 systemd-udevd[572]: /usr/lib/udev/rules.d/50-udev-default.rules:102 Unknown group 'disk', ignoring.
Dec 01 09:20:22 systemd-udevd[572]: /usr/lib/udev/rules.d/50-udev-default.rules:111 Unknown group 'kvm', ignoring.
Dec 01 09:20:22 systemd-udevd[572]: /usr/lib/udev/rules.d/50-udev-default.rules:116 Unknown group 'kvm', ignoring.
Dec 01 09:20:22 systemd-udevd[572]: /usr/lib/udev/rules.d/50-udev-default.rules:118 Unknown group 'kvm', ignoring.
Dec 01 09:20:22 systemd-udevd[572]: /usr/lib/udev/rules.d/50-udev-default.rules:120 Unknown group 'kvm', ignoring.

I did manually copy some groups over

from /usr/etc/passwd /usr/etc/group to their equivalent files in /etc

as suggested in the FAQ when I first set up the system on F40. Not sure if that is what made the difference

@sgallag-insta
Copy link
Collaborator

(For the record, I did confirm these groups do in fact exist)

@rehashedsalt
Copy link
Collaborator

So if you overlay a package that provides users or groups to bone stock f41:

  • That package's users and groups go into /usr/lib/{group,passwd}
  • Dropins go into /usr/lib/sysusers.d
  • nsswitch picks up on those files and provides identities to the system
  • systemd-sysusers doesn't do anything because it can getent the users/groups
  • System works

If you overlay a package onto Wayblue 41 (I did this with httpd for testing):

  • That packages's users and groups go into /usr/lib/{group,passwd}
  • We get a dropin, but systemd-sysusers doesn't use it because nsswitch
  • System works, same as above

If we instead add a package to the bluebuild recipe into Wayblue 41:

Thus, we have two problems:

  • There's a weird discrepancy between bluebuild running rpm-ostree install during the build recipe and the end user doing the same; and
  • /etc/shadow and /etc/gshadow get poisoned once you rebase from an image that has a user/group to one that doesn't

cc @RoyalOughtness

We've got two ways forward:

  • Figure out why bluebuild is doing a weird and not adding users/groups to /usr/lib/{passwd,group}; and/or
  • Add a small script and accompanying unit to Wayblue that runs before systemd-sysusers and uses getent to clean /etc/{,g}shadow of entries that have no matches. We'd also have to decide whether an entry for a group that has members is one worth keeping or not. Those entries would prohibit group creation if those groups are readded to Wayblue and are added via the systemd-sysusers method and not /usr/lib/group.

For bluebuild I'm definitely gonna need help on this one -- it's not my forte and I have zero confidence determining if this is an issue on our end, something inherent to the way images are built and distributed, or something on bluebuild's end.

Also: this issue supersedes a workaround mentioned in #46.

@rehashedsalt rehashedsalt mentioned this issue Dec 1, 2024
Open
@rehashedsalt
Copy link
Collaborator

I've got a fix that implements the workaround drafted up, testing is going on in Discord. Once I know that it works on machines other than mine and my VMs, I'll draft a PR with a unit file and a couple dropins for unit dependencies.

@eXsoR65
Copy link

eXsoR65 commented Dec 9, 2024 •
edited
Loading

I've got a fix that implements the workaround drafted up, testing is going on in Discord. Once I know that it works on machines other than mine and my VMs, I'll draft a PR with a unit file and a couple dropins for unit dependencies.

I confirmed this is working posted in Discord: message on discord

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@rehashedsalt rehashedsalt

Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@rehashedsalt @eXsoR65 @sgallag-insta @RoyalOughtness