Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Stopping containers doesn't work on Ubuntu 23.10 #1891

Open
jfbauer432 opened this issue Dec 27, 2023 · 35 comments
Open

Stopping containers doesn't work on Ubuntu 23.10 #1891

jfbauer432 opened this issue Dec 27, 2023 · 35 comments
Labels
Priority: Medium Status: Available Type: Bug Unintended behavior

Comments

@jfbauer432
Copy link

Describe the bug

Several weeks ago I noticed errors from watchtower when updating other containers. At first I thought the container was taking too long to exit, so I added a --stop-timeout=60s to the watchtower command, but that didn't help. Looking more info the logs, I see that it gets the error right way after trying to stop the container. This happens when trying to update different containers. I can manually restart then just fine.

Not certain, but this issue might have started after upgrading from Ubuntu 23.04 to Ubuntu 23.10. Or possibly after the most recent watchtower update

REPOSITORY                     TAG           IMAGE ID       CREATED         SIZE
containrrr/watchtower          latest        e7dd50d07b86   6 weeks ago     14.7MB

Steps to reproduce

  1. docker run --rm --name watchtower -v /var/run/docker.sock:/var/run/docker.sock containrrr/watchtower --debug --run-once unifi
    (see full log below)
  2. gets error
time="2023-12-27T13:38:08Z" level=info msg="Stopping /unifi (ff08acd24e6b) with SIGTERM"
time="2023-12-27T13:38:08Z" level=error msg="Error response from daemon: Cannot kill container: ff08acd24e6b453fcbb32c8842e4f6fe73d5e1f75091bcc6896d74602ea47a82: Cannot kill container ff08acd24e6b453fcbb32c8842e4f6fe73d5e1f75091bcc6896d74602ea47a82: unknown error after kill: runc did not terminate successfully: exit status 1: unable to signal init: permission denied\n: unknown"
time="2023-12-27T13:38:08Z" level=info msg="Session done" Failed=1 Scanned=1 Updated=0 notify=no

Expected behavior

Expected that watchtower can successfully kill (and restart) containers.

Screenshots

No response

Environment

  • Platform Ubuntu 23.10
  • Architecture x86_64
  • Docker Version 24.0.5, build 24.0.5-0ubuntu1

Your logs

$ docker run --rm --name watchtower -v /var/run/docker.sock:/var/run/docker.sock containrrr/watchtower --debug --run-once unifi
time="2023-12-27T13:38:06Z" level=debug msg="Sleeping for a second to ensure the docker api client has been properly initialized."
time="2023-12-27T13:38:07Z" level=debug msg="Making sure everything is sane before starting"
time="2023-12-27T13:38:07Z" level=info msg="Watchtower 1.7.1"
time="2023-12-27T13:38:07Z" level=info msg="Using no notifications"
time="2023-12-27T13:38:07Z" level=info msg="Only checking containers which name matches \"unifi\""
time="2023-12-27T13:38:07Z" level=info msg="Running a one time update."
time="2023-12-27T13:38:07Z" level=debug msg="Checking containers for updated images"
time="2023-12-27T13:38:07Z" level=debug msg="Retrieving running containers"
time="2023-12-27T13:38:07Z" level=debug msg="Trying to load authentication credentials." container=/unifi image="jacobalberty/unifi:latest"
time="2023-12-27T13:38:07Z" level=debug msg="No credentials for index.docker.io found" config_file=/config.json
time="2023-12-27T13:38:07Z" level=debug msg="Got image name: jacobalberty/unifi:latest"
time="2023-12-27T13:38:07Z" level=debug msg="Checking if pull is needed" container=/unifi image="jacobalberty/unifi:latest"
time="2023-12-27T13:38:07Z" level=debug msg="Built challenge URL" URL="https://index.docker.io/v2/"
time="2023-12-27T13:38:07Z" level=debug msg="Got response to challenge request" header="Bearer realm=\"https://auth.docker.io/token\",service=\"registry.docker.io\"" status="401 Unauthorized"
time="2023-12-27T13:38:07Z" level=debug msg="Checking challenge header content" realm="https://auth.docker.io/token" service=registry.docker.io
time="2023-12-27T13:38:07Z" level=debug msg="Setting scope for auth token" image=docker.io/jacobalberty/unifi scope="repository:jacobalberty/unifi:pull"
time="2023-12-27T13:38:07Z" level=debug msg="No credentials found."
time="2023-12-27T13:38:07Z" level=debug msg="Parsing image ref" host=index.docker.io image=jacobalberty/unifi normalized=docker.io/jacobalberty/unifi tag=latest
time="2023-12-27T13:38:07Z" level=debug msg="Doing a HEAD request to fetch a digest" url="https://index.docker.io/v2/jacobalberty/unifi/manifests/latest"
time="2023-12-27T13:38:08Z" level=debug msg="Found a remote digest to compare with" remote="sha256:23d3114328fcb21fd7a0ae37a38d7b65b4e21adc653b58aeadc1e3aecc4b30d6"
time="2023-12-27T13:38:08Z" level=debug msg=Comparing local="sha256:250536053f6902b324e0cf181e8a4601d8bc2e94287f42bcc7999732abadd45c" remote="sha256:23d3114328fcb21fd7a0ae37a38d7b65b4e21adc653b58aeadc1e3aecc4b30d6"
time="2023-12-27T13:38:08Z" level=debug msg="Digests did not match, doing a pull."
time="2023-12-27T13:38:08Z" level=debug msg="Pulling image" container=/unifi image="jacobalberty/unifi:latest"
time="2023-12-27T13:38:08Z" level=info msg="Found new jacobalberty/unifi:latest image (ec36c92163a6)"
time="2023-12-27T13:38:08Z" level=info msg="Stopping /unifi (ff08acd24e6b) with SIGTERM"
time="2023-12-27T13:38:08Z" level=error msg="Error response from daemon: Cannot kill container: ff08acd24e6b453fcbb32c8842e4f6fe73d5e1f75091bcc6896d74602ea47a82: Cannot kill container ff08acd24e6b453fcbb32c8842e4f6fe73d5e1f75091bcc6896d74602ea47a82: unknown error after kill: runc did not terminate successfully: exit status 1: unable to signal init: permission denied\n: unknown"
time="2023-12-27T13:38:08Z" level=info msg="Session done" Failed=1 Scanned=1 Updated=0 notify=no
time="2023-12-27T13:38:08Z" level=info msg="Waiting for the notification goroutine to finish" notify=no

Additional context

No response
@github-actions GitHub Actions
Copy link

Hi there! 👋🏼 As you're new to this repo, we'd like to suggest that you read our code of conduct as well as our contribution guidelines. Thanks a bunch for opening your first issue! 🙏

@danst0
Copy link

danst0 commented Dec 30, 2023 •
edited
Loading

Same here, came after my upgrade to ubuntu 23.10.

@piksel
Copy link
Member

piksel commented Dec 31, 2023 •
edited
Loading

Yeah, seems to be related to apparmor permissions. Perhaps @simskij has some insights?

@jfbauer432
Copy link
Author

Yeah, seems to be related to apparmor permissions.

Here is a kernel log that happened when watchtower got the error

kernel: [510121.642686] audit: type=1400 audit(1703840408.407:47786): apparmor="DENIED" operation="signal" class="signal" profile="docker-default" pid=1330911 comm="runc" requested_mask="receive" denied_mask="receive" signal=term peer="/usr/sbin/runc"

@piksel
Copy link
Member

piksel commented Jan 1, 2024

This bug contains some more info (not on Ubuntu 23.10, so I cannot test it unfortunately):
https://bugs.launchpad.net/ubuntu/+source/apparmor/+bug/2039294

I guess we need to allow watchtower/docker to send signals using an apparmor rule somehow. I have very little experience with apparmor though :/

@piksel piksel changed the title Cannot kill container, runc did not terminate successfully Stopping containers doesn't work on Ubuntu 23.10 Jan 1, 2024
@jfbauer432
Copy link
Author

Looks like this change fixes the issue

diff -u -r /etc/apparmor.d.bak/docker /etc/apparmor.d/docker
--- /etc/apparmor.d.bak/docker  2024-01-05 13:51:22.718451513 -0500
+++ /etc/apparmor.d/docker      2024-01-05 13:59:18.445871143 -0500
@@ -37,4 +37,5 @@
   # suppress ptrace denials when using 'docker ps' or using 'ps' inside a container
   ptrace (trace,read) peer=docker-default,
 
+  signal receive set=(term kill) peer="/usr/sbin/runc",
 }

and then tell apparmor to reload the modified profile by running

sudo apparmor_parser -r /etc/apparmor.d/docker

@danst0
Copy link

danst0 commented Jan 6, 2024 •
edited
Loading

I don't have a /etc/apparmor.d/docker profile, according to aa-status I have a docker-default profile, however cannot find that in /etc/apparmor.d/

ps. from my short research I think the correct way would be to create a new profile and pass that to Watchtower as a security option (https://docker-docs.uclv.cu/engine/security/apparmor/)

@lukaszzyla
Copy link

lukaszzyla commented Jan 16, 2024 •
edited
Loading

same issue here - ubuntu 23.10

@Quinten0508
Copy link

Docker automatically generates and loads a default profile for containers named docker-default. On Docker versions 1.13.0 and later, the Docker binary generates this profile in tmpfs and then loads it into the kernel. On Docker versions earlier than 1.13.0, this profile is generated in /etc/apparmor.d/docker instead.

Since the default AppArmor profile is generated dynamically you will not be able to edit it, or (afaik) even read its current contents. Could anyone on an older version share their /etc/apparmor.d/docker so we can use it as template to create a modified AppArmor profile for watchtower?

@lukaszzyla
Copy link

lukaszzyla commented Jan 21, 2024 via email

@danst0
Copy link

danst0 commented Jan 21, 2024

Ok, a little bit more context would be great. what did you do? Apt reinstall docker.io did not work.

@lukaszzyla
Copy link

lukaszzyla commented Jan 21, 2024 via email

@danst0
Copy link

danst0 commented Jan 22, 2024 via email

@lukaszzyla
Copy link

lukaszzyla commented Jan 22, 2024 via email

@danst0
Copy link

danst0 commented Jan 23, 2024

There is not package docker or docker-engine. reinstall did not work. Still looking for a solution.

@lukaszzyla
Copy link

lukaszzyla commented Jan 23, 2024 via email

@AnthonySchuijlenburg
Copy link

Doing the complete reinstall (but skipping the cleanup of old containers/images/volumes) worked for me.

@danst0
Copy link

danst0 commented Jan 24, 2024

Complete uninstall, reinstall didn't work here.

@lukaszzyla
Copy link

lukaszzyla commented Jan 24, 2024 via email •
edited
Loading

@danst0
Copy link

danst0 commented Jan 24, 2024 

root@cassius # apt list apparmor
Auflistung... Fertig
apparmor/mantic,now 4.0.0~alpha2-0ubuntu5 amd64  [installiert]

Seems to be.

root@cassius4 /h/# apparmor_status | grep docker
   docker-default
   /package/admin/s6-2.11.3.2/command/s6-svscan (6499) docker-default
   /package/admin/s6-2.11.3.2/command/s6-supervise (6586) docker-default
[...]

@lukaszzyla
Copy link

lukaszzyla commented Jan 24, 2024 •
edited
Loading

I can't be of much help here.
for me apt list states:

lukasz@chw-homeserver:~$ sudo apt list apparmor
Listing... Done
apparmor/mantic,now 4.0.0~alpha2-0ubuntu5 amd64 [installed,automatic]
`
[installed,automatic] - I have no clue why and if it does make any difference...

@KingOperator
Copy link

KingOperator commented Feb 6, 2024 •
edited
Loading

Hi,
it stopped working for me since upgrading to Ubuntu 23.10 too. Reinstall didn't work as well.
Tried:

sudo apt remove docker*
sudo apt autoremove
sudo apt install docker.io docker-compose-v2

sudo apparmor_status | grep watchtower says: /watchtower (3014113) docker-default

apt list docker* apparmor* -i says:

apparmor/mantic,now 4.0.0~alpha2-0ubuntu5 amd64  [installiert]
docker-compose-v2/mantic,now 2.20.2+ds1-0ubuntu1 amd64  [installiert]
docker.io/mantic,now 24.0.5-0ubuntu1 amd64  [installiert]

The file /etc/apparmor.d/docker doesn't exist.

@LeGEC
Copy link

LeGEC commented Mar 11, 2024 •
edited
Loading

As a temporary patch on my system, I disabled the apparmor rules for /usr/sbin/runc

Following the documentation to disable one single apparmor profile (link on help.ubuntu.com: Disable one profile ) :

sudo ln -s /etc/apparmor.d/usr.sbin.runc /etc/apparmor.d/disable/
sudo apparmor_parser -R /etc/apparmor.d/usr.sbin.runc

docker can now send signals to its containers.

re-activating is documented in the next paragraph in the page above:

sudo rm /etc/apparmor.d/disable/usr.sbin.runc
sudo apparmor_parser -r /etc/apparmor.d/usr.sbin.runc

docker stop will not be able to send a signal to its containers anymore.

(repost of my comment on the launchpad issue)

note: if anyone has a better fix (something that would only target the action of runc on docker containers for example ?) I'm interested

@kprinssu
Copy link

kprinssu commented Jun 5, 2024

@LeGEC There has been some activity on the Launchpad Issue and the fixes have been merged in the upstream projects (Moby and containerd).

The fix that worked for me is to do the following:

  1. Edit or create a new file at /etc/apparmor.d/docker-default with the following contents:
#include <tunables/global>


profile docker-default flags=(attach_disconnected,mediate_deleted) {

  #include <abstractions/base>


  network,
  capability,
  file,
  umount,
  # Host (privileged) processes may send signals to container processes.
  signal (receive) peer=unconfined,
  # dockerd may send signals to container processes (for "docker kill").
  signal (receive) peer=unconfined,
  # runc may send signals to container processes
  signal (receive) peer=runc,
  # Container processes may send signals amongst themselves.
  signal (send,receive) peer=docker-default,

  deny @{PROC}/* w,   # deny write for all files directly in /proc (not in a subdir)
  # deny write to files not in /proc/<number>/** or /proc/sys/**
  deny @{PROC}/{[^1-9],[^1-9][^0-9],[^1-9s][^0-9y][^0-9s],[^1-9][^0-9][^0-9][^0-9/]*}/** w,
  deny @{PROC}/sys/[^k]** w,  # deny /proc/sys except /proc/sys/k* (effectively /proc/sys/kernel)
  deny @{PROC}/sys/kernel/{?,??,[^s][^h][^m]**} w,  # deny everything except shm* in /proc/sys/kernel/
  deny @{PROC}/sysrq-trigger rwklx,
  deny @{PROC}/kcore rwklx,

  deny mount,

  deny /sys/[^f]*/** wklx,
  deny /sys/f[^s]*/** wklx,
  deny /sys/fs/[^c]*/** wklx,
  deny /sys/fs/c[^g]*/** wklx,
  deny /sys/fs/cg[^r]*/** wklx,
  deny /sys/firmware/** rwklx,
  deny /sys/devices/virtual/powercap/** rwklx,
  deny /sys/kernel/security/** rwklx,

  # suppress ptrace denials when using 'docker ps' or using 'ps' inside a container
  ptrace (trace,read,tracedby,readby) peer=docker-default,
}
  1. Tell AppArmor to load the new profile via apparmor_parser -Kr /etc/apparmor.d/docker-default

@ocampeau
Copy link

I had a similar issue with a bare metal installation of Kubernetes. After upgrading to Ubuntu 23.10, my cluster was in a bad state and rebooting it with kubeadm failed with the same error:

runc did not terminate successfully: exit status 1: unable to signal init: permission denied

My solution was not only to re-install containerd, which is the CRI I use, but to also delete all container mounts under /var/lib/containerd/k8s/.

Re-installing containerd was not enough. I had to delete all previous containers too.

@Garfonso
Copy link

@kprinssu Thank you very much for posting the solution here. First, it did not work for me, and I was a bit frustrated because I've been searching for months. I managed to look into the apparmor message about the denial today and saw that for me the peer is not runc but, for some reason, /usr/sbin/runc which seems to make a difference. So I adjusted the line with peer=runc to peer=/usr/sbin/runc.

Yay, finally. :-)

@UladzimirTrehubenka
Copy link

UladzimirTrehubenka commented Sep 3, 2024 •
edited
Loading

The issue: SIGTERM is not catched inside a container (Mint 22).
Workaround: sudo systemctl stop apparmor && sudo systemctl disable apparmor && reboot.

@Liamlu28
Copy link

Liamlu28 commented Sep 9, 2024

It happens on Ubuntu 24.04(Kubeneters 1.30). But I never test on new versions.

ln -s /etc/apparmor.d/runc /etc/apparmor.d/disable/
apparmor_parser -R /etc/apparmor.d/runc

@atimin atimin mentioned this issue Sep 30, 2024
Closed
1 task
@Sommerbo
Copy link

Sommerbo commented Oct 7, 2024

I fixed it by running aa-remove-unknown. For some reason AppArmor's "learning" is pretty junky and sometimes learns very odd things. Or at least that is my take on it.
And to be clear this is an AppArmor flaw/fault, watchtower just happened to trigger it. I am betting reinstalling triggers a cleanup of the AppArmor rules.

@ChristophNow
Copy link

I have the same error

@4NE3Q
Copy link

4NE3Q commented Nov 20, 2024 •
edited
Loading

The issue seems to be on ubuntu release greater than 22.04. I have created a Kubernetes cluster on ubuntu 24.04 and I'm facint this issue. I do a new deployment and the previous one's pods won't terminate. Logs below if anyone can help me. My Kubernetes is on Ubuntu 24.04.

Warning FailedKillPod 3m48s (x14 over 13m) kubelet error killing pod: [failed to "KillContainer" for "redacted" with KillContainerError: "rpc error: code = Unknown desc = failed to kill container "98837f85dacd8fdce140338e77f0b8cd2439b4c0e9c901d9fc6467c24d428db0": unknown error after kill: runc did not terminate successfully: exit status 1: unable to signal init: permission denied\n: unknown", failed to "KillPodSandbox" for "67d219c5-00dc-4405-840f-148bd60d32d2" with KillPodSandboxError: "rpc error: code = Unknown desc = failed to stop container "98837f85dacd8fdce140338e77f0b8cd2439b4c0e9c901d9fc6467c24d428db0": failed to kill container "98837f85dacd8fdce140338e77f0b8cd2439b4c0e9c901d9fc6467c24d428db0": unknown error after kill: runc did not terminate successfully: exit status 1: unable to signal init: permission denied\n: unknown"]

@Garfonso
Copy link

after kill: runc did not terminate successfully: exit status 1: unable to signal init: permission denied

You probably also need to adjust the apparmor profile, like so many of us. Read this post and the linked issue. It explains what you can do.

@4NE3Q
Copy link

4NE3Q commented Nov 20, 2024

after kill: runc did not terminate successfully: exit status 1: unable to signal init: permission denied

You probably also need to adjust the apparmor profile, like so many of us. Read this post and the linked issue. It explains what you can do.

I tried that but that didn't help.

@JingshanDu
Copy link

I fixed it by running aa-remove-unknown. For some reason AppArmor's "learning" is pretty junky and sometimes learns very odd things. Or at least that is my take on it. And to be clear this is an AppArmor flaw/fault, watchtower just happened to trigger it. I am betting reinstalling triggers a cleanup of the AppArmor rules.

This quick fix worked for me. Thank you! -- Ubuntu 24.04.1 LTS

@laoshanxi
Copy link

The issue: SIGTERM is not catched inside a container (Mint 22). Workaround: sudo systemctl stop apparmor && sudo systemctl disable apparmor && reboot.

Thank you, this works for me.

@laoshanxi laoshanxi mentioned this issue Jan 8, 2025
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Priority: Medium Status: Available Type: Bug Unintended behavior
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests