- Description
- Requirements
- Installation
- Tasks
- Events
- Advanced Topics
- Known Issues
- Breaking Changes
- Release Process
This is a report processor & fact terminus for Puppet to submit data to Splunk's logging system using Splunk's HTTP Event Collector service. There is a complimentary app in SplunkBase called Puppet Report Viewer that generates useful dashboards and makes searching this data easier. The Puppet Report Viewer app should be installed in Splunk before configuring this module.
It is possible to only include data in reports based on specific conditions (Puppet Agent Run failure, compilation failure, change, etc.) See Customized-Reporting in the Advanced Topics section for details on using that.
To enable this module:
- Classify your Puppet Servers with the splunk_hec class
- Set the
url
parameter which refers to your Splunk url along with the token provided by Splunk - Set the
splunk_token
parameter which will be the HEC token you create in Splunk. - Set the
enable_reports
to true
This module sends data to Splunk by modifying your report processor settings and indirector routes.yaml.
To send Orchestrator jobs and Event activity to Splunk, follow the instructions in the Events Section.
There are two Tasks included in this module, splunk_hec:bolt_apply
and splunk_hec:bolt_result
, that provide a pre-packaged way to compose Bolt Plans that submit data to Splunk every time they are run. Example plans are included which demonstrate task usage.
- Puppet Enterprise or Open Source Puppet
- Splunk
This was tested on both Puppet Enterprise 2019.5.0 & Puppet 6, using stock gems of yaml, json, net::https
Instructions assume you are using Puppet Enterprise. For Open Source Puppet installations please see the Custom Installation page located in the Advanced Topics section.
- Install the Puppet Report Viewer app in Splunk if not already installed
- Please see Splunk Installation if you need to install splunk
- Alternatively you can install splunk via Bolt Bolt Splunk Installation
- Create an HEC token in Splunk
- Navigate to
Settings
>Data Input
in your Splunk console - Add a new
HTTP Event Collector
with a name of your choice - Ensure
indexer acknowledgement
is not enabled - Click Next and set the source type to Automatic.
- Ensure the
App Context
is set toPuppet Report Viewer
- Add the
main
index - Set the Default Index to
main
- Click Review and then Submit
When complete the hec token should look something like this
- Navigate to
- Add the class
splunk_hec
to the PE Infrastructure -> PE Masters node group under Classification- Install the
splunk_hec
module on your Puppet masterpuppet module install puppetlabs-splunk_hec --version 0.7.1
- Navigate to
Classification
and expand thePE Infrastructure
group in the PE console - Select
PE Master
and thenConfiguration
- Add the
splunk_hec
class - Enable these parameters:
enable_reports = true manage_routes = true splunk_token = something like F5129FC8-7272-442B-983C-203F013C1948 url = something like https://splunk-8.splunk.internal:8088/services/collector include_api_collection = true
- Hit save
- Run Puppet on the node group, this will cause a restart of the Puppet-Server service
- Install the
- Configure the Splunk Puppet Report Viewer with your HEC token like so
- Log into the Splunk Console, search
index=* sourcetype=puppet:summary
and if everything was done properly, you should see the reports (and soon facts) from the systems in your Puppet environment
Configuring SSL support for this report processor and tasks requires that the Splunk HEC service being used has a properly configured SSL certificate. Once the HEC service has a valid SSL certificate, the CA will need to be made available to the report processor to load. The supported path is to install a copy of the Splunk CA to a directory called /etc/puppetlabs/puppet/splunk_hec/
and provide the file name to splunk_hec
class.
One can update the splunk_hec.yaml file with these settings:
"ssl_ca" : "splunk_ca.cert"
Or create a profile that copies the splunk_ca.cert
as part of invoking the splunk_hec class:
class profile::splunk_hec {
file { '/etc/puppetlabs/puppet/splunk_hec':
ensure => directory,
owner => 'pe-puppet',
group => 'pe-puppet',
mode => 0644,
}
file { '/etc/puppetlabs/puppet/splunk_hec/splunk_ca.cert':
ensure => file,
owner => 'pe-puppet',
group => 'pe-puppet',
mode => '0644',
source => 'puppet:///modules/profile/splunk_hec/splunk_ca.cert',
}
}
As of 0.8.0 and later the report processor can be configured to include Logs and Resource Events along with the existing summary data. Because this data varies between runs and agents in Puppet, it is difficult to predict how much data one will use in Splunk as a result. However this removes the need for configuring the Detailed Report Generation alerts in Splunk to retrieve that information, which is useful for large installations that need to retrieve a large amount of data. You can now just send the information from Puppet directly.
Add one or more of these parameters based on the desired outcome, these apply to the state of the puppet runs, one cannot filter by facts on which nodes these are in effect for. So one can get logs when a puppet run fails
, but not logs when a windows server puppet run fails
. By default none of these are enabled.
Array: Determines if logs should be included based on the return status of the puppet agent run. The can be none, one, or any of the following: failed changed unchanged
Boolean: Include logs if a catalog fails to compile. This is a more specific type of failure that indicates a serverside issue. Values: true false
Boolean: Include logs if a there is a corrective change (a PE only feature) - indicating drift was detected from the last time puppet ran on the system. Values: true false
Array: Determines if resource events should be included based on the return status of the puppet agent run. Note: this only includes resources whose status is not unchanged
- not the entire catalog. The can be none, one, or any of the following: failed changed unchanged
Boolean: Include resource events if a there is a corrective change (a PE only feature) - indicating drift was detected from the last time puppet ran on the system. Values: true false
String: If include_resources_corrective_change
or include_resources_status
is set and therefore resource_events
are being sent as part of puppet:summary
events, we can choose what format they should be sent in. Depending on your usage within Splunk, different format may be preferable, the possible values are (hash
, array
). Default: hash
. Here is an example of the data that will be forwarded to splunk in each instance:
hash
{
"resource_events": {
"File[/etc/something.conf]": {
"resource": "File[/etc/something.conf]",
"failed": false,
"out_of_sync": true
}
}
}
array
{
"resource_events": [
{
"resource": "File[/etc/something.conf]",
"failed": false,
"out_of_sync": true
}
]
}
The splunk_hec module allows the posting of PE orchestrator and activity service events to splunk.
- To utilize the API collector, a user with the correct RBAC priviledges will need to be created. The User must have read access to the Orchestrator and Activity Service API's in Puppet Enterprise.
- The common events module will need to be installed. The instructions are below.
- From your PE console, set the
include_api_collection
parameter in the splunk_hec class to true. - Set up api authentication.
- Username / Password Auth.
- Set the
pe_username
parameter to a pe user with read access to the Orchestrator and Activity Service API's in Puppet Enterprise. - Set thepe_password
parameter to the password for the user above. - Token Auth.
- Set the
pe_token
parameter for a user with read access to the Orchestrator and Activity Service API's in Puppet Enterprise. - Optionally set the
pe_console
parameter if not the server this module is installed on.- Should be the fqdn of the Puppet server you wish to use. Omit the http protocol.
- Ensure you have the latest version of the Puppet Report Viewer installed (Version 3.0.2 or higher) in your Splunk installation.
- Here is the link: https://splunkbase.splunk.com/app/4413/
- This will ensure that you have the needed source types to apply to your data in Splunk.
- Configure the Splunk_hec class as described above.
- Install the Common Events module.
- Specify this module in your Puppetfile.
mod 'common_events', :git => 'https://github.com/puppetlabs/puppetlabs-common_events'
- Specify this module in your Puppetfile.
- Run
puppet agent -t
The events now will be collected and posted to your Splunk Server. These events will appear in the Splunk UI.
There are three possible source types.
puppet:jobs
puppet:activities_classifier
puppet:activities_rbac
Use source="puppet:jobs"
in your Splunk search field to show the orchestrator jobs. Orchestrator jobs includes Puppet agent runs kicked off from the Run Puppet
button in the console, and it includes Tasks and Plans run from the console using the Run Task
and Run Plan
buttons.
Use source="puppet:activities_classifier"
in your Splunk search field to show Classifier events coming from the Activity Service API. These events will include things like creating new classifier node groups, changing node group classification rules, etc.
User source="puppet:activities_rbac
in your Splunk search field to show rbac events coming from the Activity Service API. These events will include things like creating new local users, updating user metadata, etc.
The splunk_hec module also supports customizing the fact_terminus
and facts_cache_terminus
names in the custom routes.yaml it deploys. If you are using a different facts_terminus (ie, not PuppetDB), you will want to set that parameter.
If you are already using a custom routes.yaml, these are the equivalent instructions of what the splunk_hec module does. The most important setting is configuring cache: splunk_hec
- Create a custom splunk_routes.yaml file to override where facts are cached
master: facts: terminus: puppetdb cache: splunk_hec
- Set this routes file instead of the default one with
puppet config set route_file /etc/puppetlabs/puppet/splunk_routes.yaml --section master
Two tasks are provided for submitting data from a Bolt plan to Splunk. For clarity, we recommend using a different HEC token to distinguish between events from Puppet runs and those generated by Bolt. The Puppet Report Viewer addon includes a puppet:bolt sourcetype to faciltate this. Currently SSL validation for Bolt communications to Splunk is not supported.
splunk_hec::bolt_apply
: A task that uses the remote task option to submit a Bolt Apply report in a similar format to the puppet:summary. Unlike the summary, this includes the facts from a target because those are available to bolt at execution time and added to the report data before submission to Splunk.
splunk_hec::bolt_result
: A task that sends the result of a function to Splunk. Since the format is freeform and dependent on the individual function/tasks being called, formatting of the data is best done in the plan itself prior to submitting the result hash to the task.
To setup, add the splunk_hec endpoint as a remote target in inventory.yml
:
---
nodes:
- name: splunk_bolt_hec
config:
transport: remote
remote:
hostname: <hostname>
token: <token>
port: 8088
See the plans/
directory for working examples of apply and result usage.
- Custom Installation
- Advanced Puppet Configuration
- Advanced Splunk Configuration
- Customized Reporting
- Fact Terminus Support
- Puppet Metrics Collection
- SSL Support
- Troublshooting and Verification
- Running the Tests
- Integration with puppet_metrics_collection only works on version >= 6.0.0
- SSL Validation is under active development and behavior may change
- Automated testing could use work
- 0.5.0 splunk_hec::url parameter now expects a full URI of https://servername:8088/services/collector
- 0.5.0 -> 0.6.0 Switches to the fact terminus cache setting via routes.yaml to ensure compatibility with CD4PE, see Fact Terminus Support for guides on how to change it. Prior to deploying this module, remove the setting
facts_terminus
from thepuppet_enterprise::profile::master
class in thePE Master
node group in your environment if you set it in previous revisions of this module (olders than 0.6.0). It will prevent PE from operating normally if left on.
This module is hooked up with an automatic release process using github actions. To provoke a release simply check the module out locally, tag with the new release version, then github will promote the build to the forge.
Full process to prepare for a release:
Update metadata.json to reflect new module release version (0.8.1)
Run bundle exec rake changelog
to update the CHANGELOG automatically
Submit PR for changes
Create Tag on target version:
git tag -a v0.8.1 -m "0.8.1 Feature Release"
git push upstream --tags
P.I.E. Team
P. uppet
I. ntegrations
E. ngineering
Chris Barker [email protected]
Helen Campbell [email protected]
Greg Hardy [email protected]
Bryan Jen [email protected]
Greg Sparks [email protected]