Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature: system packages for Linux servers, CI and workstations #654

Closed
alexellis opened this issue Apr 1, 2022 · 19 comments
Closed

Feature: system packages for Linux servers, CI and workstations #654

alexellis opened this issue Apr 1, 2022 · 19 comments

Comments

@alexellis
Copy link
Owner

alexellis commented Apr 1, 2022

Feature

Feature: system packages for Linux servers and workstations

There are so many times that I log into a server and install server software using a combination of curl and tar.

This provides the fastest way to install software, ensures the latest versions are available, something that package managers struggle to keep up with.

Generally, during scripting and automation, you'll find this kind of pattern. The popularity of arkade get has shown that people are comfortable installing software in this way, and it makes CI - trivial - https://github.com/hopefulramen/k3sup-tester/blob/master/.github/workflows/build.yml

For example:

arkade system install [email protected]
arkade system install [email protected]
arkade system install [email protected]
arkade system install [email protected]
arkade system install cni-plugins

This is present in openfaas/faasd, and on every Linux workstation I use, I tend to need these tools.

When a tool is already present, installation will be skipped, unless --force is passed.

The version will be detected the same way as arkade get without using the GitHub API, but using a HTTP HEAD request to the releases page, where no version is given that is.

A Go template will be used to form the URL, which will have a unit test for ARM32, ARM64 and AMD64.

Sometimes there are additional steps to run like systemd enable, or making a directory in a known location.

This would be a new command aimed at ARM32, ARM64 and AMD/Intel workstations and servers, and would save a lot of time for CI and setting up workstations.

I'd suggest we create a basic fluid SDK when adding the first app for Golang, for instance:

export ARCH=$(uname -p)

export GOPATH=$HOME/go/
export PATH=$PATH:/usr/local/go/bin/

install_go() {
    DL_ARCH=$ARCH
    if [ "$ARCH" == "x86_64" ]; then
        DL_ARCH=amd64
    else 
        if [ "$ARCH" == "aarch64" ]; then
            DL_ARCH=arm64
        fi
    fi

    if ! [ -e /usr/local/go/bin/go ]; then

        curl -SLf https://go.dev/dl/go$GOVER.linux-$DL_ARCH.tar.gz > /tmp/go.tgz
        sudo rm -rf /usr/local/go/
        sudo mkdir -p /usr/local/go/
        sudo tar -xvf /tmp/go.tgz -C /usr/local/go/ --strip-components=1

        go version
        echo "export GOPATH=\$HOME/go/" | tee -a $HOME/.bashrc
        echo "export PATH=\$PATH:/usr/local/go/bin/" | tee -a $HOME/.bashrc
    fi
}

Would become:

system.
  WithTarget("/usr/local/go/).
  WithArch("x86_64").
  WithOverwrite(false).
  WithVersion("v1.18.1")
  WithMessageTemplate(`
# Add to your PATH variable or .bashrc:
export GOPATH=\$HOME/go/" | tee -a $HOME/.bashrc
echo "export PATH=\$PATH:/usr/local/go/bin/" | tee -a $HOME/.bashrc`)

Each command file would be added and populatet the Arch / Version etc in a similar way to how we do this for arkade get

Containerd is more involved:

export CONTAINERD_VERSION=1.6.1

install_containerd() {
    DL_ARCH=$ARCH
    if [ "$ARCH" == "x86_64" ]; then
        DL_ARCH=amd64
    else 
        if [ "$ARCH" == "aarch64" ]; then
            DL_ARCH=arm64
        fi
    fi

    mkdir -p /etc/containerd

    if ! [ -e /etc/containerd/config.toml ]; then
        cd $START_PWD
        cp ./config.toml /etc/containerd/config.toml
    fi

    if ! [ -e /usr/local/bin/containerd ]; then
        curl -sSL https://github.com/containerd/containerd/releases/download/v$CONTAINERD_VERSION/containerd-$CONTAINERD_VERSION-linux-$DL_ARCH.tar.gz > /tmp/containerd.tar.gz \
        && sudo tar -xvf /tmp/containerd.tar.gz -C /usr/local/bin/ --strip-components=1

        containerd -version

        curl -sLS https://raw.githubusercontent.com/containerd/containerd/v1.5.4/containerd.service > /tmp/containerd.service

        echo "[Manager]" | tee -a /tmp/containerd.service
        echo "DefaultTimeoutStartSec=3m" | tee -a /tmp/containerd.service

        sudo cp /tmp/containerd.service /lib/systemd/system/
        sudo systemctl enable containerd || echo "Failed to enable containerd, may be masked."

        sudo systemctl daemon-reload
        sudo systemctl restart containerd
    fi
}

Therefore we'd want some additional SDK methods, as per the method we use for faasd: https://github.com/openfaas/faasd/blob/2885bb0c514a403d317b93e6d8add1ad52239a13/pkg/systemd/systemd.go

.WithFetch("https://raw.githubusercontent.com/containerd/containerd/v1.5.4/containerd.service", "/tmp/containerd.service")
.WithCp("/tmp/containerd.service", "/lib/systemd/system")
.WithDaemonReload().
.WithRestart("containerd")

I'm looking for someone to volunteer to add the first app, it needn't use an SDK for the first pass, we can extract this as we add apps.

An initial list of apps:

  • Prometheus
  • Golang
  • containerd
  • firecracker
  • nodejs
  • CNI plugins

I do not see this list becoming much larger than the above, but am open to suggestions from the community.

@alexellis alexellis changed the title Feature: system packages for Linux servers and workstations Feature: system packages for Linux servers, CI and workstations Apr 1, 2022
@richardcase
Copy link
Contributor

I could use this in a number of places. Using linux package managers is ok but not all the tools we use are available or they are in a different repo or an older version. Having something to download the binary would be great.

@iximiuz
Copy link

iximiuz commented Apr 2, 2022

This would be definitely helpful! I was checking if arkade could install Go on my system a couple of days ago - having arkade already used to install a bunch of tools (kind, kubectl, etc.), it felt absolutely natural to employ arkade as a replacement of a package manager too.

My only concern is about the secureness of this procedure. Pulling random stuff from the Internet becomes more and more dangerous these days :) So, maybe having some trusted registry with package signatures is a good idea. But this might defeat the sole purpose of having a lightweight and up-to-date installation mechanism.

@alexellis
Copy link
Owner Author

alexellis commented Apr 3, 2022

@iximiuz "pulling stuffed from some random place on the internet"

Explain how you got that from the proposal?

This is designed to be no different to arkade get, or the way that many people already install Go, containerd, etc.

@iximiuz
Copy link

iximiuz commented Apr 4, 2022

@alexellis Sure! When we do things like wget https://go. dev/dl/go1.18.linux-amd64.tar.gz, it does look like pulling random stuff from the Internet to me. Even though in this particular case the source is (more or less) trusted.

Validating the source of the artifact is an important step during an installation of a package, but having a checksum verification is probably an even more important step. But all these things must be transparent. When I use a package manager like apt, I kinda offload this trust to the maintainers of the system repositories at the expense of having (quite) outdated versions of software installed. But if I were to start installing packages with arkade, I would need a crystal clear understanding of:

  • From where a package comes from
  • What validations (e.g. checksums) arkade does before installing a package.

And yes, arkade get can also benefit from something similar to that.

@alexellis
Copy link
Owner Author

Whilst I agree that apt and pacman etc use GPG or other signing keys, I wouldn't have considered Google's download server to be untrusted or random. I checked, and the installation guide for Go follows the same pattern, they don't even supply a command to verify some checksum. https://go.dev/doc/install, I can't even find checksums on their download page: https://go.dev/dl/

Given that you use arkade get today, which downloads over HTTPS, it doesn't seem like a blocker? Is your main concern that you believe that TLS encryption could be broken and intercepted?

@Shikachuu
Copy link
Contributor

When we talk about Prometheus in this case, are we talking about prometheus itself only or with "standard" exporters like node_exporter as well?

@Shikachuu
Copy link
Contributor

I do not see this list becoming much larger than the above, but am open to suggestions from the community.

@alexellis may I suggest adding buildkit to the list?

alexellis added a commit that referenced this issue Apr 5, 2022
Part of #654, tested on an Intel x86_64 machine.

Signed-off-by: Alex Ellis (OpenFaaS Ltd) <[email protected]>
alexellis added a commit that referenced this issue Apr 5, 2022
Part of #654, tested on an Intel x86_64 machine.

Signed-off-by: Alex Ellis (OpenFaaS Ltd) <[email protected]>
@alexellis
Copy link
Owner Author

@Shikachuu prometheus = the download called prometheus

buildkit - potentially. I haven't had a need to add it stand-alone on a machine, and doesn't installing Docker add it?

Keep suggesting things that you personally install on Linux servers/workstations/CI hosts.

@alexellis alexellis pinned this issue Apr 5, 2022
@Shikachuu
Copy link
Contributor

Shikachuu commented Apr 5, 2022

@Shikachuu prometheus = the download called prometheus

buildkit - potentially. I haven't had a need to add it stand-alone on a machine, and doesn't installing Docker add it?

Keep suggesting things that you personally install on Linux servers/workstations/CI hosts.

buildkit itself doesn't require Docker, only runc, crun or containerd.

I acutally use buildkit with containerd and nerdctl to substitue docker on my workstation, my laptop and on my "build" server as well.
(Also if you download the nerdctl-full package from their github release, it contains this combination of tools too.)

@alexellis
Copy link
Owner Author

alexellis commented Apr 6, 2022

Completed:

  • firecracker
  • Golang
  • prometheus
  • containerd
  • actions-runner
  • node
  • cni-plugins (alias cni)

@dirien
Copy link
Contributor

dirien commented Apr 6, 2022

Really good idea, but I may suggest to start directly with cosign signed packages and show that its signed (or not signed). Think that would help tremendously!

@alexellis
Copy link
Owner Author

These tools are not signed with Cosign, so I'm not sure what you're suggesting?

@dirien dirien mentioned this issue Apr 6, 2022
15 tasks
@dirien
Copy link
Contributor

dirien commented Apr 6, 2022

These tools are not signed with Cosign, so I'm not sure what you're suggesting?

Uhh just saw that you are right. Maybe for the future!

@alexellis
Copy link
Owner Author

alexellis commented Apr 9, 2022

@dirien thanks for the PR for Prometheus, could you or @Shikachuu please send in a PR where you make the "arch" value come from a flag if .Changed is true - default it to ""? See how we for it for the get command. We need this for all of the system apps and it helps with testing.

@alexellis
Copy link
Owner Author

@Shikachuu for the containerd app, please make sure you install the systemd service as per the code I DM'd you on Discord. Then put the systemd behaviour behind a flag like --systemd=true (default true)

See also: https://github.com/openfaas/faasd/blob/2885bb0c514a403d317b93e6d8add1ad52239a13/pkg/systemd/systemd.go

@dirien
Copy link
Contributor

dirien commented Apr 10, 2022

@dirien thanks for the PR for Prometheus, could you or @Shikachuu please send in a PR where you make the "arch" value come from a flag if .Changed is true - default it to ""? See how we for it for the get command. We need this for all of the system apps and it helps with testing.

@alexellis Like this #673?

@alexellis
Copy link
Owner Author

That is exactly what I had in mind, could you cover the other apps too?

alexellis added a commit that referenced this issue Apr 10, 2022
Part of the changes in: #654

Tested on a Linux AMD64 host, downloaded as expected.

Signed-off-by: Alex Ellis (OpenFaaS Ltd) <[email protected]>
alexellis added a commit that referenced this issue Apr 10, 2022
Part of the changes in: #654

Tested on a Linux AMD64 host, downloaded as expected.

Signed-off-by: Alex Ellis (OpenFaaS Ltd) <[email protected]>
@dirien
Copy link
Contributor

dirien commented Apr 10, 2022

Hi @alexellis, added the node support and I saw you added the the CNI. 👍

alexellis added a commit that referenced this issue Jun 29, 2022
Part of #654, tested on an Intel x86_64 machine.

Signed-off-by: Alex Ellis (OpenFaaS Ltd) <[email protected]>
alexellis added a commit that referenced this issue Jun 29, 2022
Part of the changes in: #654

Tested on a Linux AMD64 host, downloaded as expected.

Signed-off-by: Alex Ellis (OpenFaaS Ltd) <[email protected]>
@alexellis
Copy link
Owner Author

We now have all the original apps requested along with actions-runner to get the latest GitHub Actions runner version.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

5 participants