diff --git a/docs/en/observability/apm/configure/outputs/logstash.asciidoc b/docs/en/observability/apm/configure/outputs/logstash.asciidoc index 0766285025..66800273c0 100644 --- a/docs/en/observability/apm/configure/outputs/logstash.asciidoc +++ b/docs/en/observability/apm/configure/outputs/logstash.asciidoc @@ -129,36 +129,42 @@ In addition to `@metadata`, APM Server provides other potentially useful fields, `data_stream` field, which can be used to conditionally operate on {observability-guide}/apm-data-model.html[event types], namespaces, or datasets. -As an example, you might want to use {ls} to route all `metric` events to the same custom metrics data stream, -rather than to service-specific data streams: +As an example, you might want to use {ls} to route all `metrics` events to the same custom metrics data stream, +rather than to service-specific data streams. -["source","json",subs="attributes"] +However, if when you combine all `metrics` events there are events that have the `data_stream.dataset` field +set to different values, indexing will fail with a message stating that the field does not accept any other values. +For example, the error might say something like `failed to parse field [data_stream.dataset] of type [constant_keyword]` +or `[constant_keyword] field [data_stream.dataset] only accepts values that are equal to the value defined in the mappings`. +This is because the `data_stream.dataset` field's mapping is set to `constant_keyword`, which expects all values of the +fields in the index to be the same. + +To prevent losing data due to failed indexing, add a {logstash-ref}/plugins-filters-mutate.html[Logstash mutate filter] +to update the value of `data_stream.dataset`. Then, you can send all metrics events to one custom metrics data stream: + +[source,json] ---- -output { +filter { if [@metadata][beat] == "apm-server" { <1> if [data_stream][type] == "metrics" { <2> - elasticsearch { - index => "%{[data_stream][type]}-custom-%{[data_stream][namespace]}" <3> - action => "create" <4> - cloud_id => "${CLOUD_ID}" <5> - cloud_auth => "${CLOUD_AUTH}" <5> - } - } else { - elasticsearch { - data_stream => "true" <6> - cloud_id => "${CLOUD_ID}" - cloud_auth => "${CLOUD_AUTH}" + mutate { + update => { "[data_stream][dataset]" => "custom" } <3> } } } } +output { + elasticsearch { + data_stream => "true" + cloud_id => "${CLOUD_ID}" <4> + cloud_auth => "${CLOUD_AUTH}" <4> + } +} ---- -<1> Only apply this output if the data is being sent from the APM Server -<2> Determine if the event type is `metric` -<3> If the event type is `metric`, output to a custom data stream: `metrics-custom-` -<4> You must explicitly set `action` to `create when using {ls} to output an index to a data stream -<5> In this example, `cloud_id` and `cloud_auth` are stored as {logstash-ref}/environment-variables.html[environment variables] -<6> For all other event types, index data directly into the predefined APM data steams +<1> Only apply this output if the data is being sent from the APM Server. +<2> Determine if the event type is `metrics`. +<3> Add a Logstash mutate filter to update the value of `data_stream.dataset`. +<4> In this example, `cloud_id` and `cloud_auth` are stored as {logstash-ref}/environment-variables.html[environment variables]. [float] === Compatibility