Skip to content

Commit

Permalink
attempt to address feedback
Browse files Browse the repository at this point in the history
  • Loading branch information
colleenmcginnis committed Sep 26, 2024
1 parent ab5a221 commit 21652ce
Showing 1 changed file with 29 additions and 18 deletions.
47 changes: 29 additions & 18 deletions docs/en/observability/apm/configure/outputs/logstash.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -129,30 +129,40 @@ In addition to `@metadata`, APM Server provides other potentially useful fields,
`data_stream` field, which can be used to conditionally operate on
{observability-guide}/apm-data-model.html[event types], namespaces, or datasets.

// TO DO:
// It's unclear in the issue whether it's _only_ for datasets or if it's
// event types, namespaces, and datasets.
If using the `data_stream` field to conditionally operate on a dataset,
you must create a <<apm-custom-index-template,`@custom` component template>>
to override that field's mapping from `constant_keyword` to `keyword`.

As an example, you might want to use {ls} to route all `metrics` events to the same custom metrics data stream,
rather than to service-specific data streams:
rather than to service-specific data streams.

However, if when you combine all `metrics` events there are events that have the `data_stream.dataset` field
set to different values, indexing will fail with a message stating that the field does not accept any other values.
This is because the `data_stream.dataset` field's mapping is set to `constant_keyword`,
which expects all values of the fields in the index to be the same.

To prevent losing data due to failed indexing, you should either:

* Create a `@custom` component template to override the fields' mapping from `constant_keyword` to `keyword`.
Read more in <<apm-custom-index-template>>.
* Add a {logstash-ref}/plugins-filters-mutate.html[Logstash mutate filter] to update the value of `data_stream.dataset`.
This option is shown in the example below.

Then, you can send all metrics events to one custom metrics data stream:

["source","json",subs="attributes"]
----
output {
if [@metadata][beat] == "apm-server" { <1>
if [data_stream][type] == "metrics" { <2>
elasticsearch {
index => "%{[data_stream][type]}-custom-%{[data_stream][namespace]}" <3>
action => "create" <4>
cloud_id => "${CLOUD_ID}" <5>
cloud_auth => "${CLOUD_AUTH}" <5>
mutate {
update => { "[data_stream][dataset]" => "custom" }
} <3>
index => "%{[data_stream][type]}-custom-%{[data_stream][namespace]}" <4>
action => "create" <5>
cloud_id => "${CLOUD_ID}" <6>
cloud_auth => "${CLOUD_AUTH}" <6>
}
} else {
elasticsearch {
data_stream => "true" <6>
data_stream => "true" <7>
cloud_id => "${CLOUD_ID}"
cloud_auth => "${CLOUD_AUTH}"
}
Expand All @@ -161,11 +171,12 @@ output {
}
----
<1> Only apply this output if the data is being sent from the APM Server
<2> Determine if the event type is `metric`
<3> If the event type is `metric`, output to a custom data stream: `metrics-custom-<YOUR_NAMESPACE>`
<4> You must explicitly set `action` to `create when using {ls} to output an index to a data stream
<5> In this example, `cloud_id` and `cloud_auth` are stored as {logstash-ref}/environment-variables.html[environment variables]
<6> For all other event types, index data directly into the predefined APM data steams
<2> Determine if the event type is `metrics`
<3> Add a Logstash mutate filter to update the value of `data_stream.dataset`.
<4> If the event type is `metrics`, output to a custom data stream: `metrics-custom-<YOUR_NAMESPACE>`
<5> You must explicitly set `action` to `create when using {ls} to output an index to a data stream
<6> In this example, `cloud_id` and `cloud_auth` are stored as {logstash-ref}/environment-variables.html[environment variables]
<7> For all other event types, index data directly into the predefined APM data steams

[float]
=== Compatibility
Expand Down

0 comments on commit 21652ce

Please sign in to comment.