Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Service collector - failed to query all services / The data is invalid #1883

Open
Outek opened this issue Feb 11, 2025 · 4 comments
Open

Service collector - failed to query all services / The data is invalid #1883

Outek opened this issue Feb 11, 2025 · 4 comments
Labels

Comments

@Outek
Copy link

Outek commented Feb 11, 2025

Current Behavior

The collector for the services fails. In the eventlog, i can see message like these:

time=2025-02-11T08:50:56.421Z level=WARN source=collect.go:212 msg="collector service failed after 5.3143ms, resulting in 0 metrics" err="failed to query all services: The data is invalid."

We have configured a config.yml file and exclude most of the services.

This is the config file

scrape:
  timeout-margin: 0.5
telemetry:
  max-requests: 5
  path: /metrics
log:
  level: warn
collector:
  scheduled_task:
    exclude: /Microsoft/.+
  service:
    include: sshd|Certificate Enrollment System|WdNisSvc|WinDefend|Splunkd|osqueryd
collectors:
  enabled: cpu,cpu_info,diskdrive,license,logical_disk,logon,memory,net,os,pagefile,scheduled_task,service,system,terminal_services,update,textfile
web:
  listen-address: :9182

Computer\HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\windows_exporter ImagePath:

"C:\Program Files\windows_exporter\windows_exporter.exe" --config.file="C:\Program Files\windows_exporter\config.yml"

Expected Behavior

The collector should not crash.

Steps To Reproduce

Environment

  • windows_exporter Version: 0.30.2
  • Windows Server Version: Several versions.. Windows Server 2019/2022/2025 Windows 10 and 11

windows_exporter logs

time=2025-02-11T08:50:56.421Z level=WARN source=collect.go:212 msg="collector service failed after 5.3143ms, resulting in 0 metrics" err="failed to query all services: The data is invalid."

Anything else?

No response

@Outek Outek added the 🐞 bug label Feb 11, 2025
@Outek Outek changed the title Service collector - failed to query all services Service collector - failed to query all services / The data is invalid Feb 11, 2025
@jkroepke
Copy link
Member

Excluding services does not have any effect here. Because the Windows API does not support filter on query, the filter is more or less a client sided solution.

Based on the Microsoft documentation: https://learn.microsoft.com/de-de/windows/win32/api/winsvc/nf-winsvc-enumservicesstatusw

The error may occurs from an underlying registry function. I also found https://answers.microsoft.com/en-us/windows/forum/all/diagnostic-policy-service-not-running-error-13-the/4650d362-e4f9-4262-b495-8058ac8fe616 which indicates that there is invalid data in your windows registry.

@Outek
Copy link
Author

Outek commented Feb 12, 2025

Well, it was working with version 0.28.1. We didn't have any issues with it.
Since v0.30.2 most of the time the configuration is working fine, after some time, the service crashes and the error appears. Restart the service fix the issue for about 1-2h then the error appears again. Sound like an overflow to me.. 500 servers are affected. We have to downgrade back to 0.28.1

@jkroepke
Copy link
Member

It would help to know, if the problem appears with 0.30.1 as well.

v0.30.2 contains a change that re-uses the memory buffer instead creating a buffer of each scrape.

@Outek
Copy link
Author

Outek commented Feb 12, 2025

Sure, thank you for the support. I try to test it as soon as possible.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants