Losing data with multiple HTTP headers with same key

[jacowessels]

Good day,

I am running into an issue with the HTTP input filter. We would like the full list of X-Forwarder-For IP's from the front-end to our device accessible in the header map. We are receiving HTTP messages from an HaProxy device that adds it's IP to the requests. However, HaProxy does not edit the existing X-Forwarded-For header by appending it's IP to the value of the existing header - it purely adds a new X-Forwarded-For header to the request. That means those requests that come into Logstash have more than one header with the same value (X-Forwarded-For.) As far as I can tell this is, in fact, allowed according to the HTTP specifications (RFC7230) since this case (multiple headers with the same key) can be treated by the intermediary device as a singular header with a comma-separated list as the value. This has been raised on HaProxy's GitHub (haproxy/haproxy#44) and they are insisting that they are in fact doing it correctly according to spec.

As it stands I believe the Logstash Http input plugin just takes whatever the last value it for that Header key. Would it be possible to check for duplicate headers in input and if a header exists to add the value to the existing item in the hashmap (comma-separated?)

Details:
Logstash 6.6.0 running in Docker container.

Config file (input section):
input{
http {
port => 12211
add_field => {
"facility" => "qos"
}
}
}

Regards,
Jaco