Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix system module with both filesets enabled #41381

Open
wants to merge 6 commits into
base: main
Choose a base branch
from

Conversation

belimawr
Copy link
Contributor

@belimawr belimawr commented Oct 22, 2024

Proposed commit message

The system module did not define an ID at the root of the config, that made the V2 input loader only start the first journald input it saw because they both ended up with the same identifier (type, ID and path). This is fixed by defining an ID at the root of the configuration templates.

The journald input now also adds the input_id key to its loggers and a non-fatal error is now logged at debug level.

The system-logs input is now marked as experimental instead of stable.

Fix lint warnings by moving toJournalConfig to input_linux.go

Move TestSystemLogsCanUseLogInput to a file without the
linux build constraint so it can run on all OSes supported by the
system integration.

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • [ ] I have made corresponding changes to the documentation
  • [ ] I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • [ ] I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

## Disruptive User Impact
## Author's Checklist

How to test this PR locally

  1. Run Filebeat with the following filebeat.yml and modules.d/system.yml (adjust credentials/addresses as necessary)

    filebeat.yml

    filebeat.inputs:
      - type: journald
        id: my-journald-normal-input
        tags:
           - journald-input
      - type: filestream
        id: my-filestream-id
        paths:
          - /tmp/flog.log
    
    filebeat.config.modules:
      path: ${path.config}/modules.d/*.yml
      reload.enabled: false
      reload.period: 1s
    
    setup.template:
      settings:
        index.number_of_shards: 1
    
    setup.kibana:
      host: "http://kibana:5601"
      username: admin
      password: testing
      ssl.verification_mode: none
    
    output.elasticsearch:
      hosts: ["http://elasticsearch:9200"]
      preset: latency
      protocol: "http"
    
      username: admin
      password: testing
      ssl.verification_mode: none

    modules.d/system.yml

    - module: system
      syslog:
        enabled: true
        var.use_journald: true
        input:
          tags:
            - from-journald
    
      auth:
        enabled: true
        var.use_journald: true
        var.tags:
          - from-journald

  2. Go to Discover in Kibana, filter by tags: from-journald

  3. Look at fileset.name from the events, make sure auth and syslog are there

Related issues

## Use cases
## Screenshots
## Logs

@belimawr belimawr added bug Team:Elastic-Agent-Data-Plane Label for the Agent Data Plane team backport-8.x Automated backport to the 8.x branch with mergify backport-8.16 Automated backport with mergify labels Oct 22, 2024
@belimawr belimawr self-assigned this Oct 22, 2024
@belimawr belimawr requested a review from a team as a code owner October 22, 2024 20:48
@elasticmachine
Copy link
Collaborator

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

@botelastic botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels Oct 22, 2024
@belimawr belimawr force-pushed the 41378-fix-system-module-only-starts-one-input branch from 0765233 to 0501772 Compare October 22, 2024 22:35
@belimawr belimawr changed the title Fix system module with both filesets enables Fix system module with both filesets enabled Oct 22, 2024
@belimawr belimawr force-pushed the 41378-fix-system-module-only-starts-one-input branch from 0501772 to 59b7cb6 Compare October 22, 2024 23:37
@pierrehilbert pierrehilbert requested review from rdner and mauri870 and removed request for khushijain21 and VihasMakwana October 23, 2024 08:42
Copy link
Member

@mauri870 mauri870 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't see the linter change nor the comment for the journald stdout, did you forgot to push?

Copy link
Contributor

mergify bot commented Oct 23, 2024

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b 41378-fix-system-module-only-starts-one-input upstream/41378-fix-system-module-only-starts-one-input
git merge upstream/main
git push upstream 41378-fix-system-module-only-starts-one-input

@belimawr belimawr force-pushed the 41378-fix-system-module-only-starts-one-input branch from 984d3c6 to 59974d2 Compare October 23, 2024 20:42
The system module did not define an ID at the root of the config, that
made the V2 input loader only start the first journald input it saw
because they both ended up with the same identifier (type, ID and
path). This is fixed by defining an ID at the root of the
configuration templates.

The journald input now also adds the `input_id` key to its loggers
and a non-fatal error is now logged at debug level.

The system-logs input is now marked as experimental instead of stable.
Fix lint warnings by moving toJournalConfig to input_linux.go
Fix TestSystemLogsCanUseLogInput and move it to a file without the
linux build constraint so it can run on all OSes supported by the
system integration.
@belimawr belimawr force-pushed the 41378-fix-system-module-only-starts-one-input branch from 59974d2 to d25f496 Compare October 23, 2024 21:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport-8.x Automated backport to the 8.x branch with mergify backport-8.16 Automated backport with mergify bug Team:Elastic-Agent-Data-Plane Label for the Agent Data Plane team
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[System module] Only one instance of Journald runs when both syslog and auth filesets are enabled
4 participants