Nextcloud All-In-One fails to update, watchtover segfaults?

[lvets]

Describe the bug

I've installed Nextcloud AIO on a docker rootless setup. I'm trying to update the mastercontainer by stopping all containers and then clicking "Update mastercontainer". This shows "Mastercontainer update currently running. It will restart the mastercontainer soon which will make it unavailable for a moment. Please wait until that's done. (Logs)" for a couple of seconds then goes back to the main AIO screen. Nothing seems to happen...

Apparently, the actual update is done by a watchtower container. When I check the logs of that container, I see the following:

time="2024-03-03T22:07:00Z" level=debug msg="Sleeping for a second to ensure the docker api client has been properly initialized."
time="2024-03-03T22:07:01Z" level=debug msg="Making sure everything is sane before starting"
time="2024-03-03T22:07:01Z" level=info msg="Watchtower 1.7.1"
time="2024-03-03T22:07:01Z" level=info msg="Using no notifications"
time="2024-03-03T22:07:01Z" level=info msg="Only checking containers which name matches \"nextcloud-aio-mastercontainer\""
time="2024-03-03T22:07:01Z" level=info msg="Running a one time update."
time="2024-03-03T22:07:01Z" level=debug msg="Checking containers for updated images"
time="2024-03-03T22:07:01Z" level=debug msg="Retrieving running containers"
time="2024-03-03T22:07:01Z" level=error msg="Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?"
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x30 pc=0x9fa4e2]

goroutine 1 [running]:
github.com/containrrr/watchtower/pkg/metrics.NewMetric({0x0, 0x0})
	/home/runner/work/watchtower/watchtower/pkg/metrics/metrics.go:31 +0x22
github.com/containrrr/watchtower/cmd.runUpdatesWithNotifications(0xc00031f490)
	/home/runner/work/watchtower/watchtower/cmd/root.go:377 +0x171
github.com/containrrr/watchtower/cmd.Run(0xc00031a300?, {0xc000332080?, 0x4?, 0x4?})
	/home/runner/work/watchtower/watchtower/cmd/root.go:168 +0x658
github.com/spf13/cobra.(*Command).execute(0xc00031a300, {0xc000138010, 0x4, 0x4})
	/home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:987 +0xaa7
github.com/spf13/cobra.(*Command).ExecuteC(0xc00031a300)
	/home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:1115 +0x417
github.com/spf13/cobra.(*Command).Execute(...)
	/home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:1039
github.com/containrrr/watchtower/cmd.Execute()
	/home/runner/work/watchtower/watchtower/cmd/root.go:75 +0x52
main.main()
	/home/runner/work/watchtower/watchtower/main.go:13 +0x17

Steps to reproduce

See above.

Expected behavior

Nextcloud All-In-One updates successfully and everything is hunky-dory.

Screenshots

No response

Environment

• Platform: Ubuntu 22.04 LTS.
• Architecture: x86-64
• Docker Version: 25.0.3.

Your logs

time="2024-03-03T22:07:00Z" level=debug msg="Sleeping for a second to ensure the docker api client has been properly initialized."
time="2024-03-03T22:07:01Z" level=debug msg="Making sure everything is sane before starting"
time="2024-03-03T22:07:01Z" level=info msg="Watchtower 1.7.1"
time="2024-03-03T22:07:01Z" level=info msg="Using no notifications"
time="2024-03-03T22:07:01Z" level=info msg="Only checking containers which name matches \"nextcloud-aio-mastercontainer\""
time="2024-03-03T22:07:01Z" level=info msg="Running a one time update."
time="2024-03-03T22:07:01Z" level=debug msg="Checking containers for updated images"
time="2024-03-03T22:07:01Z" level=debug msg="Retrieving running containers"
time="2024-03-03T22:07:01Z" level=error msg="Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?"
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x30 pc=0x9fa4e2]

goroutine 1 [running]:
github.com/containrrr/watchtower/pkg/metrics.NewMetric({0x0, 0x0})
	/home/runner/work/watchtower/watchtower/pkg/metrics/metrics.go:31 +0x22
github.com/containrrr/watchtower/cmd.runUpdatesWithNotifications(0xc00031f490)
	/home/runner/work/watchtower/watchtower/cmd/root.go:377 +0x171
github.com/containrrr/watchtower/cmd.Run(0xc00031a300?, {0xc000332080?, 0x4?, 0x4?})
	/home/runner/work/watchtower/watchtower/cmd/root.go:168 +0x658
github.com/spf13/cobra.(*Command).execute(0xc00031a300, {0xc000138010, 0x4, 0x4})
	/home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:987 +0xaa7
github.com/spf13/cobra.(*Command).ExecuteC(0xc00031a300)
	/home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:1115 +0x417
github.com/spf13/cobra.(*Command).Execute(...)
	/home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:1039
github.com/containrrr/watchtower/cmd.Execute()
	/home/runner/work/watchtower/watchtower/cmd/root.go:75 +0x52
main.main()
	/home/runner/work/watchtower/watchtower/main.go:13 +0x17

Additional context

No response

[github-actions]

Hi there! 👋🏼 As you're new to this repo, we'd like to suggest that you read our code of conduct as well as our contribution guidelines. Thanks a bunch for opening your first issue! 🙏

[DarkAxi0m]

I know this is a month ago, but @lvets did you get anywhere with this?

[Sourdface]

I just tried to update my Nextcloud AIO on a Raspberry PI 5 with docker in rootless mode and I am having the exact same problem.

[Sourdface]

The error mentions /var/run/docker.sock, which is the location of the Docker socket when running in normal (rootful) mode. This path doesn't exist if the docker daemon is running in rootless mode, and instead the path is something like /run/user/1000/docker.sock, which should be indicated by the DOCKER_HOST environment variable. This leads me to believe that something is failing to acknowledge DOCKER_HOST, or else DOCKER_HOST is not propagating properly. I tried to work around it by creating a symlink from the latter to the former, but I still get exactly the same error. Maybe symlinks don't work with unix sockets, or maybe the path in question is inside of a virtual filesystem that isn't being mapped correctly?

[Sourdface]

This appears to be the original ticket for this issue on the Nextcloud AIO issue tracker: nextcloud/all-in-one#4309

FWIW I am not yet fully convinced that the source of this issue is with Watchtower based on the limited evidence presented so far.

[lvets]

Sorry for not answering sooner, but I fixed this last weekend.

What I think happened is that at some point I (or something) started or restarted the Nextcloud AIO mastercontainer without the --env=WATCHTOWER_DOCKER_SOCKET_PATH=/run/user/1001/docker.sock parameter and this environment variable is then not propagated to the watchtower container leading to the initial error.

The fix for me (ymmv!) was to use runlike for both the mastercontainer and then the watchtower container.
There's an example at https://github.com/nextcloud/all-in-one/blob/main/manual-upgrade.md for something different, but it at least shows you how to use runlike.
Some changes from the top of my mind:

• Don't use sudo as we're running rootless.
• Don't use /var/run/docker.sock:/var/run/docker.sock but /run/user/1000/docker.sock:/var/run/docker.sock
• Use nextcloud-aio-mastercontainer instead of nextcloud-aio-nextcloud.
• Obviously don't change the image nor the php stuff etc.

Once both mastercontainer and watchtower are restarted with runlike, the upgrade works again.

[Sourdface]

If the issue is that the watchtower container is being started with the wrong parameters, then that is not the fault of watchtower itself; it was just using the socket path it was told to use, which doesn't exist. I think this issue should be reported as part of Nextcloud AIO. @szaimen

[Sourdface]

I was able to resolve the issue by deleting the mastercontainer instance and then re-running the initial creation command. In my case the command looks like this because I have a datadir mapped to a subdirectory of my home directory.

docker run \
   --init \
   --sig-proxy=false \
   --name nextcloud-aio-mastercontainer \
   --restart always \
   --publish 80:80 \
   --publish 8080:8080 \
   --publish 8443:8443 \
   --volume nextcloud_aio_mastercontainer:/mnt/docker-aio-config \
   --volume "$XDG_RUNTIME_DIR/docker.sock":/var/run/docker.sock:ro \
   --env NEXTCLOUD_DATADIR="$HOME/nextcloud/data/nc" \
   --env WATCHTOWER_DOCKER_SOCKET_PATH="$XDG_RUNTIME_DIR/docker.sock" \
   nextcloud/all-in-one:latest \
 ;

The instructions for rootless Docker do explicitly mention needing to set WATCHTOWER_DOCKER_SOCKET_PATH to ensure updates work, but I don't know whether my original command had that env var set or not. It seems likely that that line just got omitted, in which case this is just user error on my part and not a problem with either Watchtower or Nextcloud.