Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[APM] Add note about @custom component templates #4200

Merged
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
48 changes: 27 additions & 21 deletions docs/en/observability/apm/configure/outputs/logstash.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -129,36 +129,42 @@ In addition to `@metadata`, APM Server provides other potentially useful fields,
`data_stream` field, which can be used to conditionally operate on
{observability-guide}/apm-data-model.html[event types], namespaces, or datasets.

As an example, you might want to use {ls} to route all `metric` events to the same custom metrics data stream,
rather than to service-specific data streams:
As an example, you might want to use {ls} to route all `metrics` events to the same custom metrics data stream,
rather than to service-specific data streams.

["source","json",subs="attributes"]
However, if when you combine all `metrics` events there are events that have the `data_stream.dataset` field
set to different values, indexing will fail with a message stating that the field does not accept any other values.
For example, the error might say something like `failed to parse field [data_stream.dataset] of type [constant_keyword]`
or `[constant_keyword] field [data_stream.dataset] only accepts values that are equal to the value defined in the mappings`.
This is because the `data_stream.dataset` field's mapping is set to `constant_keyword`, which expects all values of the
fields in the index to be the same.

To prevent losing data due to failed indexing, add a {logstash-ref}/plugins-filters-mutate.html[Logstash mutate filter]
to update the value of `data_stream.dataset`. Then, you can send all metrics events to one custom metrics data stream:

[source,json]
----
output {
filter {
if [@metadata][beat] == "apm-server" { <1>
if [data_stream][type] == "metrics" { <2>
elasticsearch {
index => "%{[data_stream][type]}-custom-%{[data_stream][namespace]}" <3>
action => "create" <4>
cloud_id => "${CLOUD_ID}" <5>
cloud_auth => "${CLOUD_AUTH}" <5>
}
} else {
elasticsearch {
data_stream => "true" <6>
cloud_id => "${CLOUD_ID}"
cloud_auth => "${CLOUD_AUTH}"
mutate {
update => { "[data_stream][dataset]" => "custom" } <3>
}
}
}
}
output {
elasticsearch {
data_stream => "true"
cloud_id => "${CLOUD_ID}" <4>
cloud_auth => "${CLOUD_AUTH}" <4>
}
}
----
colleenmcginnis marked this conversation as resolved.
Show resolved Hide resolved
<1> Only apply this output if the data is being sent from the APM Server
<2> Determine if the event type is `metric`
<3> If the event type is `metric`, output to a custom data stream: `metrics-custom-<YOUR_NAMESPACE>`
<4> You must explicitly set `action` to `create when using {ls} to output an index to a data stream
<5> In this example, `cloud_id` and `cloud_auth` are stored as {logstash-ref}/environment-variables.html[environment variables]
<6> For all other event types, index data directly into the predefined APM data steams
<1> Only apply this output if the data is being sent from the APM Server.
<2> Determine if the event type is `metrics`.
<3> Add a Logstash mutate filter to update the value of `data_stream.dataset`.
<4> In this example, `cloud_id` and `cloud_auth` are stored as {logstash-ref}/environment-variables.html[environment variables].

[float]
=== Compatibility
Expand Down