[APM] Add note about @custom component templates (#4200) (#4363)

* add note about custom component templates

* attempt to address feedback

* apply suggestions from @axw

(cherry picked from commit 15dde46)

Co-authored-by: Colleen McGinnis <[email protected]>

docs/en/observability/apm/configure/outputs/logstash.asciidoc

@@ -129,36 +129,42 @@ In addition to `@metadata`, APM Server provides other potentially useful fields,
 `data_stream` field, which can be used to conditionally operate on
 {observability-guide}/apm-data-model.html[event types], namespaces, or datasets.
 
-As an example, you might want to use {ls} to route all `metric` events to the same custom metrics data stream,
-rather than to service-specific data streams:
+As an example, you might want to use {ls} to route all `metrics` events to the same custom metrics data stream,
+rather than to service-specific data streams.
 
-["source","json",subs="attributes"]
+However, if when you combine all `metrics` events there are events that have the `data_stream.dataset` field
+set to different values, indexing will fail with a message stating that the field does not accept any other values.
+For example, the error might say something like `failed to parse field [data_stream.dataset] of type [constant_keyword]`
+or `[constant_keyword] field [data_stream.dataset] only accepts values that are equal to the value defined in the mappings`.
+This is because the `data_stream.dataset` field's mapping is set to `constant_keyword`, which expects all values of the
+fields in the index to be the same.
+
+To prevent losing data due to failed indexing, add a {logstash-ref}/plugins-filters-mutate.html[Logstash mutate filter]
+to update the value of `data_stream.dataset`. Then, you can send all metrics events to one custom metrics data stream:
+
+[source,json]
 ----
-output {
+filter {
   if [@metadata][beat] == "apm-server" { <1>
     if [data_stream][type] == "metrics" { <2>
-      elasticsearch {
-        index => "%{[data_stream][type]}-custom-%{[data_stream][namespace]}" <3>
-        action => "create" <4>
-        cloud_id => "${CLOUD_ID}" <5>
-        cloud_auth => "${CLOUD_AUTH}" <5>
-      }
-    } else {
-      elasticsearch {
-        data_stream => "true" <6>
-        cloud_id => "${CLOUD_ID}"
-        cloud_auth => "${CLOUD_AUTH}"
+      mutate {
+        update => { "[data_stream][dataset]" => "custom" } <3>
       }
     }
   }
 }
+output {
+  elasticsearch {
+    data_stream => "true"
+    cloud_id => "${CLOUD_ID}" <4>
+    cloud_auth => "${CLOUD_AUTH}" <4>
+  }
+}
 ----
-<1> Only apply this output if the data is being sent from the APM Server
-<2> Determine if the event type is `metric`
-<3> If the event type is `metric`, output to a custom data stream: `metrics-custom-<YOUR_NAMESPACE>`
-<4> You must explicitly set `action` to `create when using {ls} to output an index to a data stream
-<5> In this example, `cloud_id` and `cloud_auth` are stored as {logstash-ref}/environment-variables.html[environment variables]
-<6> For all other event types, index data directly into the predefined APM data steams
+<1> Only apply this output if the data is being sent from the APM Server.
+<2> Determine if the event type is `metrics`.
+<3> Add a Logstash mutate filter to update the value of `data_stream.dataset`.
+<4> In this example, `cloud_id` and `cloud_auth` are stored as {logstash-ref}/environment-variables.html[environment variables].
 
 [float]
 === Compatibility