[apm] Update APM docs to reflect changes related to the Elasticsearch apm-data plugin

docs/en/observability/apm/configure/general.asciidoc

@@ -173,7 +173,8 @@ Defaults to `debug/vars`.
 === Data stream namespace
 
 Change the default namespace.
-This setting changes the name of the integration's data stream.
+// Should this reference to "integration" be removed?
+This setting changes the name of the data stream.
 
 For {fleet}-managed users, the namespace is inherited from the selected {agent} policy.
 

docs/en/observability/apm/getting-started-apm/get-started-with-apm-server-binary.asciidoc

@@ -165,69 +165,6 @@ See <<apm-running-on-docker, Running on Docker>> for deploying Docker containers
 [[apm-server-configuration]]
 == Step 2: Set up and configure
 
-// This content is reused in the upgrading guide
-// tag::why-apm-integration[]
-Starting in version 8.0.0, {fleet} uses the APM integration to set up and manage APM index templates,
-{ilm-init} policies, and ingest pipelines. APM Server will only send data to {es} _after_ the APM integration has been installed.
-// end::why-apm-integration[]
-
-[float]
-=== Install the APM integration
-
-// This content is reused in the upgrading guide
-// tag::install-apm-integration[]
-[%collapsible%open]
-.**If you have an internet connection**
-====
-An internet connection is required to install the APM integration via the {fleet} UI in {kib}.
-
-// lint ignore elastic-agent
-. Open {kib} and select **Add integrations** > **Elastic APM**.
-. Click **APM integration**.
-. Click **Add Elastic APM**.
-. Click **Save and continue**.
-. Click **Add Elastic Agent later**. You do not need to run an {agent} to complete the setup.
-====
-
-// tag::install-apm-integration-no-internet[]
-[%collapsible]
-.**If you don't have an internet connection**
-====
-If your environment has network traffic restrictions, there are other ways to install the APM integration.
-See {fleet-guide}/air-gapped.html[Air-gapped environments] for more information.
-
-Option 1: Update `kibana.yml`::
-+
-Update `kibana.yml` to include the following, then restart {kib}.
-+
-[source,yaml]
-----
-xpack.fleet.packages:
-- name: apm
-  version: latest
-----
-+
-See {kibana-ref}/settings.html[Configure Kibana] to learn more about how to edit the Kibana configuration file.
-
-Option 2: Use the {fleet} API::
-+
-Use the {fleet} API to install the APM integration. To be successful, this needs to be run against the {kib}
-API, not the {es} API.
-+
-["source","yaml",subs="attributes"]
-----
-POST kbn:/api/fleet/epm/packages/apm/{apm_server_version}
-{ "force": true }
-----
-+
-See {kibana-ref}/api.html[Kibana API] to learn more about how to use the Kibana APIs.
-====
-// end::install-apm-integration-no-internet[]
-// end::install-apm-integration[]
-
-[float]
-=== Configure APM
-
 Configure APM by editing the `apm-server.yml` configuration file.
 The location of this file varies by platform--see the <<apm-directory-layout>> for help locating it.
 

docs/en/observability/apm/getting-started-apm/get-started-with-fleet-apm-server.asciidoc

@@ -32,7 +32,7 @@ include::{observability-docs-root}/docs/en/observability/tab-widgets/add-apm-int
 An internet connection is required to install the APM integration via the Fleet UI in Kibana.
 
 --
-include::{observability-docs-root}/docs/en/observability/apm/getting-started-apm/get-started-with-apm-server-binary.asciidoc[leveloffset=+2,tag=install-apm-integration-no-internet]
+include::{observability-docs-root}/docs/en/observability/apm/shared/install-apm-integration-no-internet.asciidoc[]
 --
 ****
 

docs/en/observability/apm/getting-started-apm/index.asciidoc

@@ -12,9 +12,8 @@ The {es} Service is available on AWS, GCP, and Azure.
 *To get started in minutes, follow the steps in <<get-started-with-fleet-apm-server>>.*
 ****
 
-// TODO: MOVE THIS
-IMPORTANT: Starting in version 8.0.0, {fleet} uses the APM integration to set up and manage APM index templates,
-{ilm-init} policies, and ingest pipelines. APM Server will only send data to {es} _after_ the APM integration has been installed.
+IMPORTANT: Starting in version 8.15.0, the {es} apm-data plugin manages APM index templates,
+{ilm-init} policies, and ingest pipelines.
 
 The APM Server receives performance data from your APM agents,
 validates and processes it, and then transforms the data into {es} documents.

docs/en/observability/apm/manage-storage/data-streams.asciidoc

@@ -20,7 +20,7 @@ See the {fleet-guide}/data-streams.html[{fleet} and {agent} Guide] to learn more
 
 // tag::data-streams[]
 APM data follows the `<type>-<dataset>-<namespace>` naming scheme.
-The `type` and `dataset` are predefined by the APM integration,
+The `type` and `dataset` are predefined by the {es} apm-data plugin,
 but the `namespace` is your opportunity to customize how different types of data are stored in {es}.
 There is no recommendation for what to use as your namespace--it is intentionally flexible.
 For example, you might create namespaces for each of your environments,

docs/en/observability/apm/manage-storage/ilm-how-to.asciidoc

@@ -134,7 +134,7 @@ This is not considered a breaking change as index management should continually
 [id="apm-data-streams-custom-policy{append-legacy}"]
 == Configure a custom index lifecycle policy
 
-When the APM integration is installed, {fleet} creates a default `*@custom` component template for each data stream.
+The {es} apm-data plugin creates a default `*@custom` component template for each data stream.
 The easiest way to configure a custom index lifecycle policy per data stream is to edit this template.
 
 This tutorial explains how to apply a custom index lifecycle policy to the `traces-apm` data stream.
@@ -143,8 +143,8 @@ This tutorial explains how to apply a custom index lifecycle policy to the `trac
 [id="apm-data-streams-custom-one{append-legacy}"]
 == Step 1: View data streams
 
-The **Data Streams** view in {kib} shows you the data streams,
-index templates, and index lifecycle policies associated with a given integration.
+The **Data Streams** view in {kib} shows you data streams,
+index templates, and index lifecycle policies:
 
 . Navigate to **{stack-manage-app}** > **Index Management** > **Data Streams**.
 . Search for `traces-apm` to see all data streams associated with APM trace data.

docs/en/observability/apm/manage-storage/ingest-pipelines.asciidoc

@@ -19,7 +19,7 @@ The default APM pipelines are defined in {es} apm-data plugin index templates.
 [id="custom-ingest-pipelines{append-legacy}"]
 == Custom ingest pipelines
 
-The Elastic APM integration supports custom ingest pipelines.
+Elastic APM supports custom ingest pipelines.
 A custom pipeline allows you to transform data to better match your specific use case.
 This can be useful, for example, to ensure data security by removing or obfuscating sensitive information.
 

docs/en/observability/apm/shared-kibana-endpoint.asciidoc

@@ -1,6 +1,5 @@
 // tag::shared-kibana-config[]
-APM Server uses the APM integration to set up and manage APM templates, policies, and pipelines.
-To confirm the integration is installed, APM Server polls either {es} or {kib} on startup.
+APM Server uses the {es} apm-data plugin to set up and manage APM templates, policies, and pipelines.
 When using a non-{es} output, APM Server requires access to {kib} via the
 <<apm-setup-kibana-endpoint,{kib} endpoint>>.
 

docs/en/observability/apm/shared/install-apm-integration-no-internet.asciidoc

@@ -0,0 +1,32 @@
+[%collapsible]
+.**If you don't have an internet connection**
+====
+If your environment has network traffic restrictions, there are other ways to install the APM integration.
+See {fleet-guide}/air-gapped.html[Air-gapped environments] for more information.
+
+Option 1: Update `kibana.yml`::
++
+Update `kibana.yml` to include the following, then restart {kib}.
++
+[source,yaml]
+----
+xpack.fleet.packages:
+- name: apm
+  version: latest
+----
++
+See {kibana-ref}/settings.html[Configure Kibana] to learn more about how to edit the Kibana configuration file.
+
+Option 2: Use the {fleet} API::
++
+Use the {fleet} API to install the APM integration. To be successful, this needs to be run against the {kib}
+API, not the {es} API.
++
+["source","yaml",subs="attributes"]
+----
+POST kbn:/api/fleet/epm/packages/apm/{apm_server_version}
+{ "force": true }
+----
++
+See {kibana-ref}/api.html[Kibana API] to learn more about how to use the Kibana APIs.
+====

docs/en/observability/apm/troubleshooting/common-problems.asciidoc

@@ -13,7 +13,6 @@ This section describes common problems you might encounter when using APM Server
 
 *APM UI*:
 
-* <<no-apm-data-found>>
 * <<troubleshooting-too-many-transactions>>
 * <<troubleshooting-unknown-route>>
 * <<troubleshooting-fields-unsearchable>>
@@ -158,75 +157,6 @@ it is possible to see a significant increase in OS page cache memory usage due t
 If you see a drop in throughput and excessive disk activity after enabling tail-based sampling,
 please ensure that there is enough memory headroom in the system for OS page cache to perform disk IO efficiently.
 
-[float]
-[[no-apm-data-found]]
-== Data doesn't appear in the APM UI
-
-This section can help with any of the following:
-
-* Data isn't displaying in the APM UI
-* Data isn't displaying in the APM UI after an upgrade
-* You see a message like "No Services Found",
-* You see errors like "Fielddata is disabled on text fields by default..."
-
-These problems are likely to be caused by a missing index template or ingest pipeline.
-By default, {fleet} sets up these and other APM assets when the APM integration is installed.
-Try reinstalling the APM integration by navigating to
-**Integrations** → **Elastic APM** → **Manage in Fleet** → **Settings** → **Reinstall Elastic APM**.
-
-Because assets cannot be applied to indices retroactively,
-after reinstalling the APM integration you must either wait for the index to rollover or force a rollover.
-To force a rollover, use the {ref}/indices-rollover-index.html[rollover API] to target the relevant {apm-guide-ref}/apm-data-streams.html[APM data streams].
-
-[float]
-[[apm-data-indexed-no-apm]]
-=== Data is indexed but doesn't appear in the APM UI
-
-The APM UI relies on index mappings to query and display data.
-If your APM data isn't showing up in the APM UI, but is elsewhere in {kib}, like the Discover app,
-you may have a missing index mapping.
-
-You can determine if a field was mapped correctly with the `_mapping` API.
-For example, run the following command in the {kib} {kibana-ref}/console-kibana.html[console].
-This will display the field data type of the `service.name` field.
-
-[source,curl]
-----
-GET *apm*/_mapping/field/service.name
-----
-
-If the `mapping.name.type` is `"text"`, your APM indices were not set up correctly.
-
-[source,yml]
-----
-".ds-metrics-apm.transaction.1m-default-2023.04.12-000038": {
-   "mappings": {
-      "service.name": {
-         "full_name": "service.name",
-         "mapping": {
-            "name": {
-               "type": "text" <1>
-            }
-         }
-      }
-   }
-}
-----
-<1> The `service.name` `mapping.name.type` would be `"keyword"` if this field had been set up correctly.
-
-To fix this problem, install the APM integration by following these steps:
-
-include::{observability-docs-root}/docs/en/observability/apm/getting-started-apm/get-started-with-apm-server-binary.asciidoc[leveloffset=+2,tag=install-apm-integration]
-
-This will reinstall the APM index templates and trigger a data stream index rollover.
-
-You can verify the correct index templates were installed by running the following command in the {kib} console:
-
-[source,curl]
-----
-GET /_index_template/traces-apm
-----
-
 [float]
 [[troubleshooting-too-many-transactions]]
 == Too many unique transaction names
@@ -318,7 +248,7 @@ You can also use the agent's public API to manually set a name for the transacti
 == Fields are not searchable
 
 In Elasticsearch, index templates are used to define settings and mappings that determine how fields should be analyzed.
-The recommended index templates for APM are installed by {fleet} when the Elastic APM integration is installed.
+The recommended index templates for APM come from the built-in {es} apm-data plugin.
 These templates, by default, enable and disable indexing on certain fields.
 
 As an example, some APM agents store cookie values in `http.request.cookies`.
@@ -334,9 +264,6 @@ If you don't, the data view doesn't exist.
 To fix this, navigate to the APM UI in {kib} and select *Add data*.
 In the APM tutorial, click *Load Kibana objects* to create the APM data view.
 
-If creating an APM data view doesn't solve the problem,
-see <<no-apm-data-found>> for further troubleshooting.
-
 *Ensure a field is searchable*
 There are two things you can do to if you'd like to ensure a field is searchable:
 

docs/en/observability/apm/upgrading-to-8.x.asciidoc

@@ -12,7 +12,7 @@ For a detailed look at what's new, see:
 === Notable APM changes
 
 * All index management has been removed from APM Server;
-{fleet} is now entirely responsible for setting up index templates, index lifecycle polices,
+the built-in {es} apm-data plugin is entirely responsible for setting up index templates, index lifecycle polices,
 and index pipelines.
 * APM Server now only writes to well-defined data streams;
 writing to classic indices is no longer supported.
@@ -78,30 +78,9 @@ and {observability} {observability-guide}/whats-new.html[What's new] content.
 The {stack} ({es} and {kib}) must be upgraded before APM Server.
 See the {stack-ref}/upgrading-elastic-stack.html[{stack} Installation and Upgrade Guide] for guidance.
 
-. **Install the APM integration via the {fleet} UI**
-+
-include::{observability-docs-root}/docs/en/observability/apm/getting-started-apm/get-started-with-apm-server-binary.asciidoc[leveloffset=+2,tag=why-apm-integration]
-+
---
-include::{observability-docs-root}/docs/en/observability/apm/getting-started-apm/get-started-with-apm-server-binary.asciidoc[leveloffset=+2,tag=install-apm-integration]
---
-
 . **Install the {version} APM Server release**
 +
 See <<apm-installing,install>> to find the command that works with your system.
-+
-[WARNING]
-====
-If you install version {version} of APM Server before installing the APM integration, you will see error logs similar to the following. You must go back and install the APM integration before data can be ingested into {es}.
-
-[source,json]
-----
-...
-{"log.level":"error","@timestamp":"2022-01-19T10:45:34.923+0800","log.logger":"beater","log.origin":{"file.name":"beater/waitready.go","file.line":62},"message":"precondition 'apm integration installed' failed: error querying Elasticsearch for integration index templates: unexpected HTTP status: 404 Not Found ({\"error\":{\"root_cause\":[{\"type\":\"resource_not_found_exception\",\"reason\":\"index template matching [traces-apm.sampled] not found\"}],\"type\":\"resource_not_found_exception\",\"reason\":\"index template matching [traces-apm.sampled] not found\"},\"status\":404}): to remediate, please install the apm integration: https://ela.st/apm-integration-quickstart","service.name":"apm-server","ecs.version":"1.6.0"}
-{"log.level":"error","@timestamp":"2022-01-19T10:45:37.461+0800","log.logger":"beater","log.origin":{"file.name":"beater/waitready.go","file.line":62},"message":"precondition 'apm integration installed' failed: error querying Elasticsearch for integration index templates: unexpected HTTP status: 404 Not Found ({\"error\":{\"root_cause\":[{\"type\":\"resource_not_found_exception\",\"reason\":\"index template matching [logs-apm.error] not found\"}],\"type\":\"resource_not_found_exception\",\"reason\":\"index template matching [logs-apm.error] not found\"},\"status\":404}): to remediate, please install the apm integration: https://ela.st/apm-integration-quickstart","service.name":"apm-server","ecs.version":"1.6.0"}
-...
-----
-====
 
 . **Review your configuration file**
 +

docs/en/observability/apm/view-and-analyze/interpret/mobile-session-explorer.asciidoc

@@ -15,7 +15,7 @@ The steps to follow are:
 
 * copy the `session.id` from the relevant document.
 * Open the Discover page.
-* Select the appropriate data view (use `APM` to search all datastreams)
+* Select the appropriate data view (use `APM` to search all data streams)
 * set filter to the copied `session.id`
 
 Here we can see the `session.id` guid in the metadata viewer in the error detail view: