protobuf tag not enough fields in MetricFamily.state

[sawyersteven]

What happened?

When running cscli metrics the following was printed:

ERRO[02-03-2024 00:22:59] crowdsec - goroutine crowdsec/ShowPrometheus crashed : protobuf tag not enough fields in MetricFamily.state:
ERRO[02-03-2024 00:22:59] please report this error to https://github.com/crowdsecurity/crowdsec/
ERRO[02-03-2024 00:22:59] stacktrace/report is written to /tmp/crowdsec-crash.128475427.txt : please join it to your issue
FATA[02-03-2024 00:22:59] crowdsec stopped

What did you expect to happen?

Metrics to print

How can we reproduce it (as minimally and precisely as possible)?

Ubuntu 24.04, the following commands:

curl https://packagecloud.io/install/repositories/crowdsec/crowdsec/script.deb.sh | os=ubuntu dist=jammy bash
apt install crowdsec
apt install crowdsec-firewall-bouncer-iptables
cscli metrics

No configs have been modified.

Anything else we need to know?

Crash log: https://pastebin.com/JgPEaZtp

Crowdsec version

$ cscli version
2024/03/02 00:27:13 version: v1.4.6-6build1-debian
2024/03/02 00:27:13 Codename: alphaga
2024/03/02 00:27:13 BuildDate: 2023-08-24_05:13:32
2024/03/02 00:27:13 GoVersion: 1.21.0
2024/03/02 00:27:13 Platform: linux
2024/03/02 00:27:13 Constraint_parser: >= 1.0, <= 2.0
2024/03/02 00:27:13 Constraint_scenario: >= 1.0, < 3.0
2024/03/02 00:27:13 Constraint_api: v1
2024/03/02 00:27:13 Constraint_acquis: >= 1.0, < 2.0

OS version

# On Linux:
$ cat /etc/os-release
# paste output here
$ uname -a
# paste output here

# On Windows:
C:\> wmic os get Caption, Version, BuildNumber, OSArchitecture
# paste output here

Enabled collections and parsers

$ cscli hub list -o raw
# paste output here

Acquisition config

``` $ cat /etc/crowdsec/acquis.yaml /etc/crowdsec/acquis.d/* labels: type: nginx --- filenames: - /var/log/auth.log - /var/log/syslog labels: type: syslog --- source: journalctl journalctl_filter: - "_SYSTEMD_UNIT=ssh.service" labels: type: syslog --- filename: /var/log/apache2/*.log labels: type: apache2 ```

Config show

$ cscli config show
Global:
   - Configuration Folder   : /etc/crowdsec
   - Data Folder            : /var/lib/crowdsec/data
   - Hub Folder             : /var/lib/crowdsec/hub
   - Simulation File        : /etc/crowdsec/simulation.yaml
   - Log Folder             : /var/log/
   - Log level              : info
   - Log Media              : file
Crowdsec:
  - Acquisition File        : /etc/crowdsec/acquis.yaml
  - Parsers routines        : 1
  - Acquisition Folder      : /etc/crowdsec/acquis.d
cscli:
  - Output                  : human
  - Hub Branch              :
  - Hub Folder              : /var/lib/crowdsec/hub
Local API Server:
  - Listen URL              : 127.0.0.1:8080
  - Profile File            : /etc/crowdsec/profiles.yaml
  - Trusted IPs:
      - 127.0.0.1
      - ::1
  - Database:
      - Type                : sqlite
      - Path                : /var/lib/crowdsec/data/crowdsec.db
      - Flush age           : 7d
      - Flush size          : 5000

Prometheus metrics

$ cscli metrics
# See above

Related custom configs versions (if applicable) : notification plugins, custom scenarios, parsers etc.

[github-actions]

@sawyersteven: Thanks for opening an issue, it is currently awaiting triage.

In the meantime, you can:

1. Check Crowdsec Documentation to see if your issue can be self resolved.
2. You can also join our Discord.
3. Check Releases to make sure your agent is on the latest version.

Details

I am a bot created to help the crowdsecurity developers manage community feedback and contributions. You can check out my manifest file to understand my behavior and what I can do. If you want to use this for your project, you can check out the BirthdayResearch/oss-governance-bot repository.

[LaurenceJJones]

Seems you may have two crowdsec binaries on the system as the version is quite outdated

which -a crowdsec

[sawyersteven]

$ which -a crowdsec
/usr/bin/crowdsec
/bin/crowdsec

I'm not sure how more than one binary got installed... This is a relatively fresh os install and everything else I run is containerized.

edit:

Running apt purge crowdsec results in both binaries being removed, but apt install crowdsec puts them both back.

[LaurenceJJones]

$ which -a crowdsec
/usr/bin/crowdsec
/bin/crowdsec

I'm not sure how more than one binary got installed... This is a relatively fresh os install and everything else I run is containerized.

edit:

Running apt purge crowdsec results in both binaries being removed, but apt install crowdsec puts them both back.

or maybe you have two cscli ? cause the version output you provided says 1.4.6

[sawyersteven]

$ which -a cscli
/usr/bin/cscli
/bin/cscli

But after an apt purge of everything and a reboot the cscli metrics command works as expected. But it still shows two paths (bin and usr/bin) for both cscli and crowdsec.

So if it runs I'll just have to do some good pen tests to make sure it is working correctly and actually blocks bad actors.

[mmetc]

In many distributions /bin is a link to /usr/bin so seeing two binaries is not a surprise.

[Asrael00md]

Check your /etc/apt/sources.list.d for the content of the relevant repository.
I do use Debian testing and the script.deb.sh created trixie instead of newer versions (like bookworm) as link to the repository.
You probably currently do see an error when running apt update for the URL.
After changing trixie -> bookworm (or relevant latest supported Ubuntu version), run
'apt update & apt upgrade'
You should see an upgrade to the latest Version of crowdsec (1.6.1*)

This fixed the cscli metrics issue for me.

[usma0118]

Facing same issue.

Tried @Asrael00md and that didn't resolve the issue

[LaurenceJJones]

Facing same issue.

Tried @Asrael00md and that didn't resolve the issue

can you provide output of:

for i in $(which -a cscli); do echo $i && $i version;done

[usma0118]

Facing same issue.
Tried @Asrael00md and that didn't resolve the issue

can you provide output of:

for i in $(which -a cscli); do echo $i && $i version;done

i have tried removing and purging.

confirmed binaries were removed after that only installed via apt.
files are not symlink

[LaurenceJJones]

Facing same issue.
Tried @Asrael00md and that didn't resolve the issue

can you provide output of:

for i in $(which -a cscli); do echo $i && $i version;done

i have tried removing and purging.

confirmed binaries were removed after that only installed via apt. files are not symlink

Seems you are using the older version within the debian/ubuntu repositories.

Could you purge the installation then follow our steps here:

https://docs.crowdsec.net/u/getting_started/installation/linux

[usma0118]

Facing same issue.
Tried @Asrael00md and that didn't resolve the issue

can you provide output of:

for i in $(which -a cscli); do echo $i && $i version;done

i have tried removing and purging.
confirmed binaries were removed after that only installed via apt. files are not symlink

Seems you are using the older version within the debian/ubuntu repositories.

Could you purge the installation then follow our steps here:

https://docs.crowdsec.net/u/getting_started/installation/linux

apt-cache returns this:

running:

similar outcome.

[LaurenceJJones]

root@ubuntu:~# apt list crowdsec
Listing... Done
crowdsec/noble-updates,noble-security 1.4.6-6ubuntu0.24.04.1 amd64
N: There is 1 additional version. Please use the '-a' switch to see it
root@ubuntu:~# curl https://install.crowdsec.net | sudo sh
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 14417  100 14417    0     0  89421      0 --:--:-- --:--:-- --:--:-- 89546
Detected operating system as ubuntu/24.
Detected apt version as 2.7.14
Checking for gpg...
Detected gpg...
Checking for curl...
Detected curl...
Scanning processes...
Scanning linux images...

Importing packagecloud gpg key...

Packagecloud gpg key imported to /etc/apt/keyrings/crowdsec_crowdsec-archive-keyring.gpg

Installing /etc/apt/sources.list.d/crowdsec_crowdsec.list...


root@ubuntu:~# apt list crowdsec
Listing... Done
crowdsec/any 1.6.3 amd64
N: There are 9 additional versions. Please use the '-a' switch to see them.

Can you ensure /etc/apt/sources.list.d/crowdsec_crowdsec.list exists and provide uname -a incase it arch dependent. I can see amd64 in previous image.

[usma0118]

root@ubuntu:~# apt list crowdsec
Listing... Done
crowdsec/noble-updates,noble-security 1.4.6-6ubuntu0.24.04.1 amd64
N: There is 1 additional version. Please use the '-a' switch to see it
root@ubuntu:~# curl https://install.crowdsec.net | sudo sh
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 14417  100 14417    0     0  89421      0 --:--:-- --:--:-- --:--:-- 89546
Detected operating system as ubuntu/24.
Detected apt version as 2.7.14
Checking for gpg...
Detected gpg...
Checking for curl...
Detected curl...
Scanning processes...
Scanning linux images...

Importing packagecloud gpg key...

Packagecloud gpg key imported to /etc/apt/keyrings/crowdsec_crowdsec-archive-keyring.gpg

Installing /etc/apt/sources.list.d/crowdsec_crowdsec.list...


root@ubuntu:~# apt list crowdsec
Listing... Done
crowdsec/any 1.6.3 amd64
N: There are 9 additional versions. Please use the '-a' switch to see them.

Can you ensure /etc/apt/sources.list.d/crowdsec_crowdsec.list exists and provide uname -a incase it arch dependent. I can see amd64 in previous image.

it was missing. adding it again did the trick.
Thanks it's working now.