Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[[processors.parser]] merge with more than one parsed metric does not work #16382

Open
1tft opened this issue Jan 8, 2025 · 1 comment
Open
Labels
bug unexpected problem or unintended behavior

Comments

@1tft
Copy link

1tft commented Jan 8, 2025

Relevant telegraf.conf

[[inputs.file]]
  files = [ "/tmp/mydata.txt" ]
  data_format = "grok"
  grok_patterns = ['''
    %{MULTILINEDATA:logmessage}
  ''']
  grok_multiline = true
  name_override = "prometheus" # same name as later parsed data from payload

  [inputs.file.tags]
    mytag = "mytagvalue"

[[processors.parser]]
  data_format = "prometheus"
  parse_fields = ["logmessage"]
  drop_original = false
  merge = "override" # Case one
  #merge = "" # Case two


[[processors.rename]]
  fieldexclude = ["logmessage"]
  [[processors.rename.replace]]
    measurement = "prometheus"
    dest = "mymeasurement"

Logs from Telegraf

na

System info

Telegraf 1.32.1

Docker

No response

Steps to reproduce

  1. Create /tmp/mydata.txt with following data:
# HELP mycount my cool counter
# TYPE mycount gauge
mycount{mylabel="labelvalue01"} 66
mycount{mylabel="labelvalue02"} 77
  1. Run config above (telegraf -config /etc/telegraf/telegraf.d/my_issue -test)
    ...

Expected behavior

Output:

mymeasurement,host=XYZ,mylabel=labelvalue01,mytag=mytagvalue mycount=66 1736354163000000000
mymeasurement,host=XYZ,mylabel=labelvalue02,mytag=mytagvalue mycount=77 1736354163000000000

All parsed metrics are merged with inital input metric.

Actual behavior

Output:

mymeasurement,host=XYZ,mylabel=labelvalue02,mytag=mytagvalue mycount=77 1736354163000000000

All metrics but one have been lost.

Additional info

For this example we can solve all issues when we parse prometheus already in inputs plugin (and not use grok).
But our real life issue is based on [[inputs.win_eventlog]], there we cant set data_format to "prometheus" inside inputs plugin.

Currently we solve this issue with merge = "" but this means all global tags are lost including "host" tag. Also tag mytag = "mytagvalue" is lost which we need for flow control because we use multiple configs on same host. Thats our main issue: We dont know how we can "transfer" our [inputs.file.tags] and agent.conf global tags to any metric generated by [[processors.parser]], without specifying duplicated manually via processors.starlark or processors.override.

@1tft 1tft added the bug unexpected problem or unintended behavior label Jan 8, 2025
@1tft
Copy link
Author

1tft commented Jan 8, 2025

FYI: Currently there also another issue #16331 open for [[processors.parser]]

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug unexpected problem or unintended behavior
Projects
None yet
Development

No branches or pull requests

1 participant