Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for logstash codecs #114

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,6 @@
## 5.2.5
- Add support for logstash codecs

## 5.2.4
- Relax dependency on http_client mixin since current major works on both

Expand Down
1 change: 1 addition & 0 deletions CONTRIBUTORS
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ Contributors:
* Kurt Hurtado (kurtado)
* Pier-Hugues Pellerin (ph)
* Richard Pijnenburg (electrical)
* Thomas Andrejak (ToToL)

Note: If you've sent us patches, bug reports, or otherwise contributed to
Logstash, and you aren't on the list above and want to be, please let us know
Expand Down
9 changes: 6 additions & 3 deletions docs/index.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ Consider this when tuning this plugin for performance.
Additionally, note that when parallel execution is used strict ordering of events is not
guaranteed!

Beware, this gem does not yet support codecs. Please use the 'format' option for now.
Beware, to use codecs, please use the 'format' option for now.

[id="plugins-{type}s-{plugin}-options"]
==== Http Output Configuration Options
Expand All @@ -47,7 +47,7 @@ This plugin supports the following configuration options plus the <<plugins-{typ
| <<plugins-{type}s-{plugin}-content_type>> |<<string,string>>|No
| <<plugins-{type}s-{plugin}-cookies>> |<<boolean,boolean>>|No
| <<plugins-{type}s-{plugin}-follow_redirects>> |<<boolean,boolean>>|No
| <<plugins-{type}s-{plugin}-format>> |<<string,string>>, one of `["json", "json_batch", "form", "message"]`|No
| <<plugins-{type}s-{plugin}-format>> |<<string,string>>, one of `["json", "json_batch", "form", "message", "codec"]`|No
| <<plugins-{type}s-{plugin}-headers>> |<<hash,hash>>|No
| <<plugins-{type}s-{plugin}-http_compression>> |<<boolean,boolean>>|No
| <<plugins-{type}s-{plugin}-http_method>> |<<string,string>>, one of `["put", "post", "patch", "delete", "get", "head"]`|Yes
Expand Down Expand Up @@ -136,6 +136,7 @@ If not specified, this defaults to the following:
* if format is "json", "application/json"
* if format is "json_batch", "application/json". Each Logstash batch of events will be concatenated into a single array and sent in one request.
* if format is "form", "application/x-www-form-urlencoded"
* if format is "codec", "text/xml"

[id="plugins-{type}s-{plugin}-cookies"]
===== `cookies`
Expand All @@ -157,7 +158,7 @@ Should redirects be followed? Defaults to `true`
[id="plugins-{type}s-{plugin}-format"]
===== `format`

* Value can be any of: `json`, `json_batch`, `form`, `message`
* Value can be any of: `json`, `json_batch`, `form`, `message`, `codec`
* Default value is `"json"`

Set the format of the http body.
Expand All @@ -171,6 +172,8 @@ into a query parameter string, e.g. `foo=bar&baz=fizz...`

If message, then the body will be the result of formatting the event according to message

If codec, then the body will be the result of executing the specified codec

Otherwise, the event is sent as json.

[id="plugins-{type}s-{plugin}-headers"]
Expand Down
12 changes: 10 additions & 2 deletions lib/logstash/outputs/http.rb
Original file line number Diff line number Diff line change
Expand Up @@ -81,14 +81,20 @@ class LogStash::Outputs::Http < LogStash::Outputs::Base
# If message, then the body will be the result of formatting the event according to message
#
# Otherwise, the event is sent as json.
config :format, :validate => ["json", "json_batch", "form", "message"], :default => "json"
config :format, :validate => ["json", "json_batch", "form", "message", "codec"], :default => "json"

# Set this to true if you want to enable gzip compression for your http requests
config :http_compression, :validate => :boolean, :default => false

config :message, :validate => :string

default :codec, :validate => :string, :default => "json"

def register
@codec.on_event do |event, payload|
payload
end

@http_method = @http_method.to_sym

# We count outstanding requests with this queue
Expand All @@ -106,6 +112,7 @@ def register
when "json" ; @content_type = "application/json"
when "json_batch" ; @content_type = "application/json"
when "message" ; @content_type = "text/plain"
when "codec" ; @content_type = "text/plain"
end
end

Expand Down Expand Up @@ -298,11 +305,12 @@ def log_failure(message, opts)

# Format the HTTP body
def event_body(event)
# TODO: Create an HTTP post data codec, use that here
if @format == "json"
LogStash::Json.dump(map_event(event))
elsif @format == "message"
event.sprintf(@message)
elsif @format == "codec"
@codec.encode(event)
elsif @format == "json_batch"
LogStash::Json.dump(event.map {|e| map_event(e) })
else
Expand Down
2 changes: 1 addition & 1 deletion logstash-output-http.gemspec
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
Gem::Specification.new do |s|
s.name = 'logstash-output-http'
s.version = '5.2.4'
s.version = '5.2.5'
s.licenses = ['Apache License (2.0)']
s.summary = "Sends events to a generic HTTP or HTTPS endpoint"
s.description = "This gem is a Logstash plugin required to be installed on top of the Logstash core pipeline using $LS_HOME/bin/logstash-plugin install gemname. This gem is not a stand-alone program"
Expand Down