Help to setting up sysbox in a Jenkins CI/CD pipeline to spin up jenkins agents that would spinup child containers

[jadesoturi]

Hi all,

Today we are running Jenkins master on bare-metal, together with YADP(YetAnotherDockerPlugin) to spin up jenkins docker agents on-demand using the JNLP protocol. This works, but we either have to run things as root or map the hosts docker.sock to the containers, add jenkins user to the docker group or see things fail. What we want is to use sysbox runtime on the host that the agent containers are created on, so that the agent container can then create the needed containers to do the build/tests as child containers that are isolated from the host, but that run as root inside so build tools like npm, maven and the like work without any issues(it is a pain at the moment, with half the builds failing unless we pass -u root as docker arguments for container creation. This would not be an issue if we knew that container is running inside a sysbox, and not directly on the host).

Sofar, I've taken the jenkins/inbound-agent image and added docker to it and set sysbox as the default runtime on the host. When doing a docker exec -it on the jenkins agent container, there is no /var/run/docker.sock there as far as I am able to see, and the build/test containers are still started on the host directly, but using sysbox as runtime(since its default).

I attempted to add systemd to start the dockerd service(using ubuntu-focal-systemd and ubuntu-focal-systemd-dockerd Dockerfiles as example), but got the error that the system was not started with systemd as PID1(even though we specify the systemd entrypoint as per examples in your Dockerfiles). After a little bit of reading it seems I need to run the container as --priviledged, but that kinda defeats the pupose of sysbox, no? Or am i totaly confused here and have something mixed up? And if so, how do I make it work with the jenkins/inbound-agent image to get the desired result(docker in docker with sysbox on host, dockerd in outer container and inner container running with root inside)? We do not want to migrate out Jenkins controler into docker and run the full pipeline in sysbox/docker. Only certain builds that would be provisioned to the agent based on labels.

Thank you in advance, if there is need for any clarification, I can post logs/configs etc :)

//jadesoturi

[ctalledo]

Hi @jadesoturi, thanks for giving Sysbox a shot.

I think I understand your scenario (thanks for the detailed explanation) and Sysbox should be able to help.

Have you seen these blogs yet? If not, please take a look:

• Jenkins + Sysbox (should be really helpful to you).
• GitLab CI + Sysbox (not Jenkins related, but can provide you some ideas).

[jadesoturi]

Hi @jadesoturi, thanks for giving Sysbox a shot.

I think I understand your scenario (thanks for the detailed explanation) and Sysbox should be able to help.

Have you seen these blogs yet? If not, please take a look:

• Jenkins + Sysbox (should be really helpful to you).
• GitLab CI + Sysbox (not Jenkins related, but can provide you some ideas).

Hi @ctalledo , and thank you for your reply.

Yes, I've seen the blog posts, and while they provide some great inspiration, we have not been able to get it to work. However, we have identified(we think) the issue to be the init.sh script that the YADP uses to spin up containers. Since this script is passed as command when creating the container, it also disregards the entrypoint set in the image(so even with systemd and dockerd installed in the image and entrypoint set to /sbin/init - they do not start and we get the error regarding systemd not being PID1). As such the dockerd inside the container does not run, but uses the mapped docker.sock from the host.

We have not been able to to modify the init.sh script to make it work yet, but if you have any ideas on how to work around it(since its been 3 years since the last update on the YAD plugin, I dont think there is much active development going on there), please do let me know :)

I'm going to try to modify the inbound-agent image such that it has dockerd preinstalled and running without systemd(no need for /sbin/init), which should work around the init.sh issue above. Will report back next week on how it goes.

//jadesoturi

[ctalledo]

Hi @jadesoturi, have you tried without systemd in the image? systemd is not really required in CI scenarios, it's mostly useful when running dev environments in sysbox containers. For example, the nestybox/alpine-docker image comes with Docker in it but without systemd, so you could create a derived image that uses the init.sh script you mentioned and in addition starts dockerd in the container.

but uses the mapped docker.sock from the host

This is usually not necessary with Sysbox containers, since the goal is to run the whole docker engine inside the sysbox container, rather than connecting to the host docker socket.

Does your config resemble one of the diagrams in the Jenkins + Sysbox blog? Or is it different?

[jadesoturi]

Hello @ctalledo ,
Attached are 2 png - this is a very simplified mockup, but the general gist is correct.

What we have:

What we want:

The problem is that the plugin in jenkins controller that is used to provision cloud agents(inbound agent in the pngs), is using a built in init.sh script. This results in the JIAC being spun up with this init.sh + args as startup command. This is necessary for the agent to work, as it passes some credentials etc as args. But this also results in any entrypoint set on the image is being ignored, such that autostarting dockerd or systemd fails. Based on the commit history of the plugin, not much is happening there the last 3 years. Here is the script in the plugin repo on Github: https://github.com/KostyaSha/yet-another-docker-plugin/blob/dc6e2b2b4b5966fdb76ae709a64a665f91899864/yet-another-docker-plugin/src/main/resources/com/github/kostyasha/yad/launcher/DockerComputerJNLPLauncher/init.sh

I will try to push a PR to modify the script to run dockerd, but again - does not seem there is much activity there - new issues are not being responded to and there are 14 PRs that are open.

As for the images with dockerd in them and trying without systemd - the problem is the same. I need to start either dockerd or systemd to start dockerd and to do that I either have to exec into the container manually, or have the init.sh script start it as you say.

So I guess this comes down to: is there a way to autostart the docker daemon inside the JIAC, without modifying the init.sh script, so it will not use the hosts docker.sock to spin up the build job containers?

//jadesoturi
TIA!

//jadesoturi

[ctalledo]

Hi @jadesoturi, apologies for the belated reply. The diagram of "What we want" looks pretty good to me, and certainly it's a common use case for Sysbox in CI environments.

But in your case it's really a question about how to override the entrypoint of the JIAC container (i.e., the init.sh) when provisioned by Jenkins.

Questions:

• Do you have any control on the docker run command that Jenkins issues to provision the JIAC container (e.g., can you override the entrypoint via the --entrypoint flag with your custom one)?

• Do you have any control on the JIAC container image (e.g., can you create a derived image that has your custom entrypoint that starts dockerd inside the container and then calls init.sh afterwards)?

[pguinet]

Hello,

I know this question is from a past time and is outdated, but since I'm having quite the same project, here I am.

I don't have an answer to your exact question but another question in return. I know this may not be what you wanted and i apologies in advance for that. Why did you choose YADP and not docker-plugin ? With sysbox on a bare-metal Docker host I can achieve what you want with docker-plugin.

[jadesoturi]

Hi guys. Sorry for not answering, should def. enable notifications on Github 🙈 .

We had to prioritise a little differently and pivot away form this, so I haven't worked on this for a while.

@pguinet YADP was a historical choice done long before I was here. Due to a bunch of custom Jenkins scripts in the pipeline here, we decided not to mess with what works(even if it was somewhat wobbly at times).

@ctalledo To answer your questions:

No. The docker run command is handled by the YADP, at least from what I remember right now. It's been a while.

As for the image: we are already hosting internally a modified image based on the jenkins-inbound-agent:

FROM jenkins/inbound-agent:latest

ARG docker_version=20.10.21

ENV docker_version="${docker_version}"

USER root

RUN apt-get update -qq && apt-get install -qqy \
      apt-transport-https \
      ca-certificates \
      curl \
 && rm -rf /var/lib/apt/lists/* \
 && update-ca-certificates -f \
 && curl -vOL "https://download.docker.com/linux/static/stable/x86_64/docker-${docker_version}.tgz" \
 && tar zxvf "docker-${docker_version}.tgz" \
 && chmod +x docker/docker \
 && mv docker/docker /usr/bin/ \
 && rm -rf docker*

USER jenkins

And the finished build is being stored in our local harbor and pulled in by the YADP plugin. But IIRC, we tried setting an entrypoint through the Dockerfile, but it got overruled by the YADP. I will have to check up on this though and get back to you.