From 20d825938083fd1f5e6128fc36b5a24e5debe17a Mon Sep 17 00:00:00 2001 From: Colleen McGinnis Date: Tue, 1 Oct 2024 12:50:05 -0500 Subject: [PATCH 01/16] update getting started docs --- .../apm/getting-started-apm-server.asciidoc | 73 +------------------ ...stall-apm-integration-no-internet.asciidoc | 32 ++++++++ 2 files changed, 35 insertions(+), 70 deletions(-) create mode 100644 docs/en/observability/apm/shared/install-apm-integration-no-internet.asciidoc diff --git a/docs/en/observability/apm/getting-started-apm-server.asciidoc b/docs/en/observability/apm/getting-started-apm-server.asciidoc index 9ccde19dbd..66ae94df46 100644 --- a/docs/en/observability/apm/getting-started-apm-server.asciidoc +++ b/docs/en/observability/apm/getting-started-apm-server.asciidoc @@ -10,10 +10,8 @@ TIP: The easiest way to get started with Elastic APM is by using our The {es} Service is available on AWS, GCP, and Azure. See <> to get started in minutes. - -// TODO: MOVE THIS -IMPORTANT: Starting in version 8.0.0, {fleet} uses the APM integration to set up and manage APM index templates, -{ilm-init} policies, and ingest pipelines. APM Server will only send data to {es} _after_ the APM integration has been installed. +IMPORTANT: Starting in version 8.15.0, the {es} apm-data plugin manages APM index templates, +{ilm-init} policies, and ingest pipelines. The APM Server receives performance data from your APM agents, validates and processes it, and then transforms the data into {es} documents. @@ -306,69 +304,6 @@ See <> for deploying Docker containers [[apm-server-configuration]] ==== Step 2: Set up and configure -// This content is reused in the upgrading guide -// tag::why-apm-integration[] -Starting in version 8.0.0, {fleet} uses the APM integration to set up and manage APM index templates, -{ilm-init} policies, and ingest pipelines. APM Server will only send data to {es} _after_ the APM integration has been installed. -// end::why-apm-integration[] - -[float] -===== Install the APM integration - -// This content is reused in the upgrading guide -// tag::install-apm-integration[] -[%collapsible%open] -.**If you have an internet connection** -==== -An internet connection is required to install the APM integration via the {fleet} UI in {kib}. - -// lint ignore elastic-agent -. Open {kib} and select **Add integrations** > **Elastic APM**. -. Click **APM integration**. -. Click **Add Elastic APM**. -. Click **Save and continue**. -. Click **Add Elastic Agent later**. You do not need to run an {agent} to complete the setup. -==== - -// tag::install-apm-integration-no-internet[] -[%collapsible] -.**If you don't have an internet connection** -==== -If your environment has network traffic restrictions, there are other ways to install the APM integration. -See {fleet-guide}/air-gapped.html[Air-gapped environments] for more information. - -Option 1: Update `kibana.yml`:: -+ -Update `kibana.yml` to include the following, then restart {kib}. -+ -[source,yaml] ----- -xpack.fleet.packages: -- name: apm - version: latest ----- -+ -See {kibana-ref}/settings.html[Configure Kibana] to learn more about how to edit the Kibana configuration file. - -Option 2: Use the {fleet} API:: -+ -Use the {fleet} API to install the APM integration. To be successful, this needs to be run against the {kib} -API, not the {es} API. -+ -["source","yaml",subs="attributes"] ----- -POST kbn:/api/fleet/epm/packages/apm/{apm_server_version} -{ "force": true } ----- -+ -See {kibana-ref}/api.html[Kibana API] to learn more about how to use the Kibana APIs. -==== -// end::install-apm-integration-no-internet[] -// end::install-apm-integration[] - -[float] -===== Configure APM - Configure APM by editing the `apm-server.yml` configuration file. The location of this file varies by platform--see the <> for help locating it. @@ -506,9 +441,7 @@ include::{observability-docs-root}/docs/en/observability/tab-widgets/add-apm-int **** An internet connection is required to install the APM integration via the Fleet UI in Kibana. --- -include::{observability-docs-root}/docs/en/observability/apm/getting-started-apm-server.asciidoc[tag=install-apm-integration-no-internet] --- +include::shared/install-apm-integration-no-internet.asciidoc[] **** ==== Step 3: Install APM agents diff --git a/docs/en/observability/apm/shared/install-apm-integration-no-internet.asciidoc b/docs/en/observability/apm/shared/install-apm-integration-no-internet.asciidoc new file mode 100644 index 0000000000..2c2e0cc74d --- /dev/null +++ b/docs/en/observability/apm/shared/install-apm-integration-no-internet.asciidoc @@ -0,0 +1,32 @@ +[%collapsible] +.**If you don't have an internet connection** +==== +If your environment has network traffic restrictions, there are other ways to install the APM integration. +See {fleet-guide}/air-gapped.html[Air-gapped environments] for more information. + +Option 1: Update `kibana.yml`:: ++ +Update `kibana.yml` to include the following, then restart {kib}. ++ +[source,yaml] +---- +xpack.fleet.packages: +- name: apm + version: latest +---- ++ +See {kibana-ref}/settings.html[Configure Kibana] to learn more about how to edit the Kibana configuration file. + +Option 2: Use the {fleet} API:: ++ +Use the {fleet} API to install the APM integration. To be successful, this needs to be run against the {kib} +API, not the {es} API. ++ +["source","yaml",subs="attributes"] +---- +POST kbn:/api/fleet/epm/packages/apm/{apm_server_version} +{ "force": true } +---- ++ +See {kibana-ref}/api.html[Kibana API] to learn more about how to use the Kibana APIs. +==== From 14b33b3c66a2d69328759e39643e42470905e8d1 Mon Sep 17 00:00:00 2001 From: Colleen McGinnis Date: Tue, 1 Oct 2024 12:56:14 -0500 Subject: [PATCH 02/16] update upgrade guide --- .../apm/upgrading-to-8.x.asciidoc | 21 ------------------- 1 file changed, 21 deletions(-) diff --git a/docs/en/observability/apm/upgrading-to-8.x.asciidoc b/docs/en/observability/apm/upgrading-to-8.x.asciidoc index 35bd9536ff..79a72b8028 100644 --- a/docs/en/observability/apm/upgrading-to-8.x.asciidoc +++ b/docs/en/observability/apm/upgrading-to-8.x.asciidoc @@ -78,30 +78,9 @@ and {observability} {observability-guide}/whats-new.html[What's new] content. The {stack} ({es} and {kib}) must be upgraded before APM Server. See the {stack-ref}/upgrading-elastic-stack.html[{stack} Installation and Upgrade Guide] for guidance. -. **Install the APM integration via the {fleet} UI** -+ -include::{observability-docs-root}/docs/en/observability/apm/getting-started-apm-server.asciidoc[tag=why-apm-integration] -+ --- -include::{observability-docs-root}/docs/en/observability/apm/getting-started-apm-server.asciidoc[tag=install-apm-integration] --- - . **Install the {version} APM Server release** + See <> to find the command that works with your system. -+ -[WARNING] -==== -If you install version {version} of APM Server before installing the APM integration, you will see error logs similar to the following. You must go back and install the APM integration before data can be ingested into {es}. - -[source,json] ----- -... -{"log.level":"error","@timestamp":"2022-01-19T10:45:34.923+0800","log.logger":"beater","log.origin":{"file.name":"beater/waitready.go","file.line":62},"message":"precondition 'apm integration installed' failed: error querying Elasticsearch for integration index templates: unexpected HTTP status: 404 Not Found ({\"error\":{\"root_cause\":[{\"type\":\"resource_not_found_exception\",\"reason\":\"index template matching [traces-apm.sampled] not found\"}],\"type\":\"resource_not_found_exception\",\"reason\":\"index template matching [traces-apm.sampled] not found\"},\"status\":404}): to remediate, please install the apm integration: https://ela.st/apm-integration-quickstart","service.name":"apm-server","ecs.version":"1.6.0"} -{"log.level":"error","@timestamp":"2022-01-19T10:45:37.461+0800","log.logger":"beater","log.origin":{"file.name":"beater/waitready.go","file.line":62},"message":"precondition 'apm integration installed' failed: error querying Elasticsearch for integration index templates: unexpected HTTP status: 404 Not Found ({\"error\":{\"root_cause\":[{\"type\":\"resource_not_found_exception\",\"reason\":\"index template matching [logs-apm.error] not found\"}],\"type\":\"resource_not_found_exception\",\"reason\":\"index template matching [logs-apm.error] not found\"},\"status\":404}): to remediate, please install the apm integration: https://ela.st/apm-integration-quickstart","service.name":"apm-server","ecs.version":"1.6.0"} -... ----- -==== . **Review your configuration file** + From 5ec626253cd9155e5b96a0d3c168e0f40ac4e919 Mon Sep 17 00:00:00 2001 From: Colleen McGinnis Date: Tue, 1 Oct 2024 12:57:58 -0500 Subject: [PATCH 03/16] fix build --- .../en/observability/apm/common-problems.asciidoc | 15 ++++++++------- 1 file changed, 8 insertions(+), 7 deletions(-) diff --git a/docs/en/observability/apm/common-problems.asciidoc b/docs/en/observability/apm/common-problems.asciidoc index 3df7179c2b..2c5fa74fe9 100644 --- a/docs/en/observability/apm/common-problems.asciidoc +++ b/docs/en/observability/apm/common-problems.asciidoc @@ -54,13 +54,14 @@ If the `mapping.name.type` is `"text"`, your APM indices were not set up correct ---- <1> The `service.name` `mapping.name.type` would be `"keyword"` if this field had been set up correctly. -To fix this problem, install the APM integration by following these steps: - --- -include::{observability-docs-root}/docs/en/observability/apm/getting-started-apm-server.asciidoc[tag=install-apm-integration] --- - -This will reinstall the APM index templates and trigger a data stream index rollover. +// TO DO: What is the fix now? +// To fix this problem, install the APM integration by following these steps: +// +// -- +// include::{observability-docs-root}/docs/en/observability/apm/getting-started-apm-server.asciidoc[tag=install-apm-integration] +// -- +// +// This will reinstall the APM index templates and trigger a data stream index rollover. You can verify the correct index templates were installed by running the following command in the {kib} console: From ed159f04c87e88b57178ef78e24c9491873b0482 Mon Sep 17 00:00:00 2001 From: Colleen McGinnis Date: Wed, 2 Oct 2024 12:53:33 -0500 Subject: [PATCH 04/16] audit mentions of the apm integration --- docs/en/observability/apm-ui/troubleshooting.asciidoc | 2 +- docs/en/observability/apm/data-streams.asciidoc | 3 ++- docs/en/observability/apm/ingest-pipelines.asciidoc | 2 +- docs/en/observability/apm/shared-kibana-endpoint.asciidoc | 5 +++-- 4 files changed, 7 insertions(+), 5 deletions(-) diff --git a/docs/en/observability/apm-ui/troubleshooting.asciidoc b/docs/en/observability/apm-ui/troubleshooting.asciidoc index 2dced7c1c2..86189c5024 100644 --- a/docs/en/observability/apm-ui/troubleshooting.asciidoc +++ b/docs/en/observability/apm-ui/troubleshooting.asciidoc @@ -142,7 +142,7 @@ You can also use the agent's public API to manually set a name for the transacti === Fields are not searchable In Elasticsearch, index templates are used to define settings and mappings that determine how fields should be analyzed. -The recommended index templates for APM are installed by {fleet} when the Elastic APM integration is installed. +The recommended index templates for APM come from the {es} apm-data plugin. These templates, by default, enable and disable indexing on certain fields. As an example, some APM agents store cookie values in `http.request.cookies`. diff --git a/docs/en/observability/apm/data-streams.asciidoc b/docs/en/observability/apm/data-streams.asciidoc index c5389aa4a8..9aff9bc402 100644 --- a/docs/en/observability/apm/data-streams.asciidoc +++ b/docs/en/observability/apm/data-streams.asciidoc @@ -20,7 +20,8 @@ See the {fleet-guide}/data-streams.html[{fleet} and {agent} Guide] to learn more // tag::data-streams[] APM data follows the `--` naming scheme. -The `type` and `dataset` are predefined by the APM integration, +// Is this right? +The `type` and `dataset` are predefined by the {es} apm-data plugin, but the `namespace` is your opportunity to customize how different types of data are stored in {es}. There is no recommendation for what to use as your namespace--it is intentionally flexible. For example, you might create namespaces for each of your environments, diff --git a/docs/en/observability/apm/ingest-pipelines.asciidoc b/docs/en/observability/apm/ingest-pipelines.asciidoc index 01353e8258..f6526a5853 100644 --- a/docs/en/observability/apm/ingest-pipelines.asciidoc +++ b/docs/en/observability/apm/ingest-pipelines.asciidoc @@ -19,7 +19,7 @@ The default APM pipelines are defined in {es} apm-data plugin index templates. [id="custom-ingest-pipelines{append-legacy}"] === Custom ingest pipelines -The Elastic APM integration supports custom ingest pipelines. +Elastic APM supports custom ingest pipelines. A custom pipeline allows you to transform data to better match your specific use case. This can be useful, for example, to ensure data security by removing or obfuscating sensitive information. diff --git a/docs/en/observability/apm/shared-kibana-endpoint.asciidoc b/docs/en/observability/apm/shared-kibana-endpoint.asciidoc index 195216122e..b713774acb 100644 --- a/docs/en/observability/apm/shared-kibana-endpoint.asciidoc +++ b/docs/en/observability/apm/shared-kibana-endpoint.asciidoc @@ -1,6 +1,7 @@ // tag::shared-kibana-config[] -APM Server uses the APM integration to set up and manage APM templates, policies, and pipelines. -To confirm the integration is installed, APM Server polls either {es} or {kib} on startup. +APM Server uses the {es} apm-data plugin to set up and manage APM templates, policies, and pipelines. +// Does it make sense to say the plugin is "installed"? +To confirm the plugin is installed, APM Server polls either {es} or {kib} on startup. When using a non-{es} output, APM Server requires access to {kib} via the <>. From a8d92758393dcb675c84552b05060dd14a95b026 Mon Sep 17 00:00:00 2001 From: Colleen McGinnis Date: Fri, 4 Oct 2024 09:37:31 -0500 Subject: [PATCH 05/16] audit mentions of data streams --- .../observability/apm-ui/mobile-session-explorer.asciidoc | 2 +- docs/en/observability/apm/configure/general.asciidoc | 1 + docs/en/observability/apm/ilm-how-to.asciidoc | 6 +++--- 3 files changed, 5 insertions(+), 4 deletions(-) diff --git a/docs/en/observability/apm-ui/mobile-session-explorer.asciidoc b/docs/en/observability/apm-ui/mobile-session-explorer.asciidoc index dc1a2fb8cb..37f34e67fa 100644 --- a/docs/en/observability/apm-ui/mobile-session-explorer.asciidoc +++ b/docs/en/observability/apm-ui/mobile-session-explorer.asciidoc @@ -15,7 +15,7 @@ The steps to follow are: * copy the `session.id` from the relevant document. * Open the Discover page. -* Select the appropriate data view (use `APM` to search all datastreams) +* Select the appropriate data view (use `APM` to search all data streams) * set filter to the copied `session.id` Here we can see the `session.id` guid in the metadata viewer in the error detail view: diff --git a/docs/en/observability/apm/configure/general.asciidoc b/docs/en/observability/apm/configure/general.asciidoc index decdc577e8..6daeab1ab5 100644 --- a/docs/en/observability/apm/configure/general.asciidoc +++ b/docs/en/observability/apm/configure/general.asciidoc @@ -173,6 +173,7 @@ Defaults to `debug/vars`. == Data stream namespace Change the default namespace. +// Should this reference to "integration" be removed? This setting changes the name of the integration's data stream. For {fleet}-managed users, the namespace is inherited from the selected {agent} policy. diff --git a/docs/en/observability/apm/ilm-how-to.asciidoc b/docs/en/observability/apm/ilm-how-to.asciidoc index 59d57bcbff..d8597e4ffd 100644 --- a/docs/en/observability/apm/ilm-how-to.asciidoc +++ b/docs/en/observability/apm/ilm-how-to.asciidoc @@ -134,7 +134,7 @@ This is not considered a breaking change as index management should continually [id="apm-data-streams-custom-policy{append-legacy}"] === Configure a custom index lifecycle policy -When the APM integration is installed, {fleet} creates a default `*@custom` component template for each data stream. +The {es} apm-data plugin creates a default `*@custom` component template for each data stream. The easiest way to configure a custom index lifecycle policy per data stream is to edit this template. This tutorial explains how to apply a custom index lifecycle policy to the `traces-apm` data stream. @@ -143,8 +143,8 @@ This tutorial explains how to apply a custom index lifecycle policy to the `trac [id="apm-data-streams-custom-one{append-legacy}"] === Step 1: View data streams -The **Data Streams** view in {kib} shows you the data streams, -index templates, and index lifecycle policies associated with a given integration. +The **Data Streams** view in {kib} shows you data streams, +index templates, and index lifecycle policies: . Navigate to **{stack-manage-app}** > **Index Management** > **Data Streams**. . Search for `traces-apm` to see all data streams associated with APM trace data. From 701c9fc305957200f47f0427be42f4a0faefac5e Mon Sep 17 00:00:00 2001 From: Colleen McGinnis Date: Fri, 4 Oct 2024 09:45:51 -0500 Subject: [PATCH 06/16] audit mentions of index templates --- docs/en/observability/apm-ui/troubleshooting.asciidoc | 2 ++ docs/en/observability/apm/upgrading-to-8.x.asciidoc | 1 + 2 files changed, 3 insertions(+) diff --git a/docs/en/observability/apm-ui/troubleshooting.asciidoc b/docs/en/observability/apm-ui/troubleshooting.asciidoc index 86189c5024..93f4dcb344 100644 --- a/docs/en/observability/apm-ui/troubleshooting.asciidoc +++ b/docs/en/observability/apm-ui/troubleshooting.asciidoc @@ -42,6 +42,8 @@ This section can help with any of the following: * You see a message like "No Services Found", * You see errors like "Fielddata is disabled on text fields by default..." +// Assuming the APM integration is no longer responsible for index templates and ingest pipelines, +// what is the fix now? These problems are likely to be caused by a missing index template or ingest pipeline. By default, {fleet} sets up these and other APM assets when the APM integration is installed. Try reinstalling the APM integration by navigating to diff --git a/docs/en/observability/apm/upgrading-to-8.x.asciidoc b/docs/en/observability/apm/upgrading-to-8.x.asciidoc index 79a72b8028..2f99b98d2b 100644 --- a/docs/en/observability/apm/upgrading-to-8.x.asciidoc +++ b/docs/en/observability/apm/upgrading-to-8.x.asciidoc @@ -11,6 +11,7 @@ For a detailed look at what's new, see: [float] === Notable APM changes +// Does the first bullet point need to be updated at all? * All index management has been removed from APM Server; {fleet} is now entirely responsible for setting up index templates, index lifecycle polices, and index pipelines. From cd5be47814c0a9795f6f58535d9c3c9119f525ec Mon Sep 17 00:00:00 2001 From: Colleen McGinnis Date: Fri, 4 Oct 2024 09:49:47 -0500 Subject: [PATCH 07/16] audit mentions of index mappings --- docs/en/observability/apm/common-problems.asciidoc | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/docs/en/observability/apm/common-problems.asciidoc b/docs/en/observability/apm/common-problems.asciidoc index 2c5fa74fe9..32d8db203a 100644 --- a/docs/en/observability/apm/common-problems.asciidoc +++ b/docs/en/observability/apm/common-problems.asciidoc @@ -54,14 +54,15 @@ If the `mapping.name.type` is `"text"`, your APM indices were not set up correct ---- <1> The `service.name` `mapping.name.type` would be `"keyword"` if this field had been set up correctly. -// TO DO: What is the fix now? -// To fix this problem, install the APM integration by following these steps: -// +// Assuming the APM integration is no longer responsible for index mappings, +// what is the fix now? +To fix this problem, install the APM integration by following these steps: + // -- // include::{observability-docs-root}/docs/en/observability/apm/getting-started-apm-server.asciidoc[tag=install-apm-integration] // -- -// -// This will reinstall the APM index templates and trigger a data stream index rollover. + +This will reinstall the APM index templates and trigger a data stream index rollover. You can verify the correct index templates were installed by running the following command in the {kib} console: From b3428019a06ebd4d013a2a3ae1007039fe3808c6 Mon Sep 17 00:00:00 2001 From: Colleen McGinnis Date: Mon, 7 Oct 2024 08:58:38 -0500 Subject: [PATCH 08/16] address feedback from @carsonip --- .../apm/configure/general.asciidoc | 2 +- .../apm/manage-storage/data-streams.asciidoc | 1 - .../apm/shared-kibana-endpoint.asciidoc | 2 - .../troubleshooting/common-problems.asciidoc | 81 +------------------ .../apm/upgrading-to-8.x.asciidoc | 3 +- 5 files changed, 3 insertions(+), 86 deletions(-) diff --git a/docs/en/observability/apm/configure/general.asciidoc b/docs/en/observability/apm/configure/general.asciidoc index 59aa2a296e..db1c5cd0e7 100644 --- a/docs/en/observability/apm/configure/general.asciidoc +++ b/docs/en/observability/apm/configure/general.asciidoc @@ -174,7 +174,7 @@ Defaults to `debug/vars`. Change the default namespace. // Should this reference to "integration" be removed? -This setting changes the name of the integration's data stream. +This setting changes the name of the data stream. For {fleet}-managed users, the namespace is inherited from the selected {agent} policy. diff --git a/docs/en/observability/apm/manage-storage/data-streams.asciidoc b/docs/en/observability/apm/manage-storage/data-streams.asciidoc index 6915805e85..09333df6c0 100644 --- a/docs/en/observability/apm/manage-storage/data-streams.asciidoc +++ b/docs/en/observability/apm/manage-storage/data-streams.asciidoc @@ -20,7 +20,6 @@ See the {fleet-guide}/data-streams.html[{fleet} and {agent} Guide] to learn more // tag::data-streams[] APM data follows the `--` naming scheme. -// Is this right? The `type` and `dataset` are predefined by the {es} apm-data plugin, but the `namespace` is your opportunity to customize how different types of data are stored in {es}. There is no recommendation for what to use as your namespace--it is intentionally flexible. diff --git a/docs/en/observability/apm/shared-kibana-endpoint.asciidoc b/docs/en/observability/apm/shared-kibana-endpoint.asciidoc index b713774acb..fbc89285fa 100644 --- a/docs/en/observability/apm/shared-kibana-endpoint.asciidoc +++ b/docs/en/observability/apm/shared-kibana-endpoint.asciidoc @@ -1,7 +1,5 @@ // tag::shared-kibana-config[] APM Server uses the {es} apm-data plugin to set up and manage APM templates, policies, and pipelines. -// Does it make sense to say the plugin is "installed"? -To confirm the plugin is installed, APM Server polls either {es} or {kib} on startup. When using a non-{es} output, APM Server requires access to {kib} via the <>. diff --git a/docs/en/observability/apm/troubleshooting/common-problems.asciidoc b/docs/en/observability/apm/troubleshooting/common-problems.asciidoc index 2a082d5d09..e31cce8bde 100644 --- a/docs/en/observability/apm/troubleshooting/common-problems.asciidoc +++ b/docs/en/observability/apm/troubleshooting/common-problems.asciidoc @@ -13,7 +13,6 @@ This section describes common problems you might encounter when using APM Server *APM UI*: -* <> * <> * <> * <> @@ -158,81 +157,6 @@ it is possible to see a significant increase in OS page cache memory usage due t If you see a drop in throughput and excessive disk activity after enabling tail-based sampling, please ensure that there is enough memory headroom in the system for OS page cache to perform disk IO efficiently. -[float] -[[no-apm-data-found]] -== Data doesn't appear in the APM UI - -This section can help with any of the following: - -* Data isn't displaying in the APM UI -* Data isn't displaying in the APM UI after an upgrade -* You see a message like "No Services Found", -* You see errors like "Fielddata is disabled on text fields by default..." - -// Assuming the APM integration is no longer responsible for index templates and ingest pipelines, -// what is the fix now? -These problems are likely to be caused by a missing index template or ingest pipeline. -By default, {fleet} sets up these and other APM assets when the APM integration is installed. -Try reinstalling the APM integration by navigating to -**Integrations** → **Elastic APM** → **Manage in Fleet** → **Settings** → **Reinstall Elastic APM**. - -Because assets cannot be applied to indices retroactively, -after reinstalling the APM integration you must either wait for the index to rollover or force a rollover. -To force a rollover, use the {ref}/indices-rollover-index.html[rollover API] to target the relevant {apm-guide-ref}/apm-data-streams.html[APM data streams]. - -[float] -[[apm-data-indexed-no-apm]] -=== Data is indexed but doesn't appear in the APM UI - -The APM UI relies on index mappings to query and display data. -If your APM data isn't showing up in the APM UI, but is elsewhere in {kib}, like the Discover app, -you may have a missing index mapping. - -You can determine if a field was mapped correctly with the `_mapping` API. -For example, run the following command in the {kib} {kibana-ref}/console-kibana.html[console]. -This will display the field data type of the `service.name` field. - -[source,curl] ----- -GET *apm*/_mapping/field/service.name ----- - -If the `mapping.name.type` is `"text"`, your APM indices were not set up correctly. - -[source,yml] ----- -".ds-metrics-apm.transaction.1m-default-2023.04.12-000038": { - "mappings": { - "service.name": { - "full_name": "service.name", - "mapping": { - "name": { - "type": "text" <1> - } - } - } - } -} ----- -<1> The `service.name` `mapping.name.type` would be `"keyword"` if this field had been set up correctly. - -// Assuming the APM integration is no longer responsible for index mappings, -// what is the fix now? -To fix this problem, install the APM integration by following these steps: - -// -- -// include::{observability-docs-root}/docs/en/observability/apm/getting-started-apm-server.asciidoc[tag=install-apm-integration] -// -- - -This will reinstall the APM index templates and trigger a data stream index rollover. - -You can verify the correct index templates were installed by running the following command in the {kib} console: - -[source,curl] ----- -GET /_index_template/traces-apm ----- - [float] [[troubleshooting-too-many-transactions]] == Too many unique transaction names @@ -324,7 +248,7 @@ You can also use the agent's public API to manually set a name for the transacti == Fields are not searchable In Elasticsearch, index templates are used to define settings and mappings that determine how fields should be analyzed. -The recommended index templates for APM come from the {es} apm-data plugin. +The recommended index templates for APM come from the built-in {es} apm-data plugin. These templates, by default, enable and disable indexing on certain fields. As an example, some APM agents store cookie values in `http.request.cookies`. @@ -340,9 +264,6 @@ If you don't, the data view doesn't exist. To fix this, navigate to the APM UI in {kib} and select *Add data*. In the APM tutorial, click *Load Kibana objects* to create the APM data view. -If creating an APM data view doesn't solve the problem, -see <> for further troubleshooting. - *Ensure a field is searchable* There are two things you can do to if you'd like to ensure a field is searchable: diff --git a/docs/en/observability/apm/upgrading-to-8.x.asciidoc b/docs/en/observability/apm/upgrading-to-8.x.asciidoc index 2f99b98d2b..06fb375d5a 100644 --- a/docs/en/observability/apm/upgrading-to-8.x.asciidoc +++ b/docs/en/observability/apm/upgrading-to-8.x.asciidoc @@ -11,9 +11,8 @@ For a detailed look at what's new, see: [float] === Notable APM changes -// Does the first bullet point need to be updated at all? * All index management has been removed from APM Server; -{fleet} is now entirely responsible for setting up index templates, index lifecycle polices, +the built-in {es} apm-data plugin is entirely responsible for setting up index templates, index lifecycle polices, and index pipelines. * APM Server now only writes to well-defined data streams; writing to classic indices is no longer supported. From d14869bf20b119693e8857960bdf3a8c75dc6b2a Mon Sep 17 00:00:00 2001 From: Colleen McGinnis Date: Mon, 7 Oct 2024 13:12:15 -0500 Subject: [PATCH 09/16] update diagram --- docs/en/observability/apm/images/bin-ov.png | Bin 103628 -> 38766 bytes 1 file changed, 0 insertions(+), 0 deletions(-) diff --git a/docs/en/observability/apm/images/bin-ov.png b/docs/en/observability/apm/images/bin-ov.png index 7702dd7d765e856af6235e0fc05e416d351690da..848294ab8234d1a23149387295a7297f4e86771d 100644 GIT binary patch literal 38766 zcmd?Qby(AX+dn)53$c(c6#-#{w6xNV(lAOIM%SbfMN~?pyPMHtun~d^(!Eh*3Je${ zBu0&R=6zjXzxVI=*Yo%De2!zs*x0+yK40hg{_tE|h3XpPH2?rWrKYN=3jkauzs^Cf zTq1wNcJ3-S`R%HQs);uMK+Eyxbxuw9;cxPn=e%`QOAOjn|KQq8;u$8oBWw1S2zjABO>2&~cO2F4 zUilvOT1}{tE4BfR%;sB9=KlZmgbhAE|Ic}A z+XW7>;l z{~a7ATH$W>3QmL6ko-aI?XvxC3nQ8$p4v01d&M{%m0C%84sy+!z(ybN<6|LDwup>^ z9(B%OFXO2A!8N+OB)AFI4dSF#*HI`@yX4rpb5lptb|*a6152ec3sH~^S|d)fBtY!F z&nrH>_|Jl{UJCs#!w9daoMl{$WwvEDOpN0_bes`}+S(T0$f>M6$C&$(Z^IWJJ;9YO z?ynJ5iw_qq2nI!}lxVYJxs0W4fNe~bIjYy5N8F?NhXg+Q*q^^0celpBZH+wH%*aZ@ ze^a8&5uabSqgrA6u~ygk>glxIn&a7*Vb!==1y4}PrC$xN*uCpLNM2qKq&NZT4^+wG z{69}6n_)3DUMWVxsZyH9v46VNU?^Jsdb61yVzaoPb!*`Cx5PrW1!adN*2=ybX7AHM zxgqQ%?LBUKW`TXx;V@PoGD@G|SB=rd4Ajc=$ls!E6b2|myg^BZCzgop$I)X7~+tMT@nzoEM>Y?Ncr5Hr`>&wLnwJ5 zRtFvG&S_IAahFP;7?LMKE0ANC8+AqZ+<&+K<)e_lgV^%vqHWCM##cX-*`gK$TV7#y zw6$#OH--$vZ(L`4Yz}D8 zgZA~5U6&?YbF%o{z2D&HtI!l%LL;s_O55454vSXrpxV25;#_9j-MXq-#SY1GL|7d1VApHv7r3 zub0DaO8Hw@+ui%iek+PA;rTmr?=>PD2QpKlWqBmhjM^h%kOQ&mmS$u30m)DUklaV{@|QtOaatM;oml$TZ%+|L z1C%OaaBDbNnp10~snR+s&|$C{mukH*qfHuYB(^U?qh?(^Gi z9sL&Yn<`|!DT65EuA%NC_WqG=t28HLpS>0d+-T~_#!JpF&?x^S>BY|SNM~3yxG?ly zJ$NaTax|JH3p`~M8M{8U;g$TRssM!6YrOldm`RJtC1Wx^pOQuNRT77_VpXDn>OPe6 zcelAm*?m_|ltW2NsCx?@H&aB#&x~P%_`7|W@tr<>{Yzn5-3XzmpkH=7Upq@$mdDup zUs*C%23;10I{|FjF$G;kn?Y%BF=R^ucL-uAP(Sw6*C4Pgg-czR}-eo^4V&2J+*? z$iJ2%%r5_2lKYz~Fp!}K@z!3abFxCr+$BfS*Lrp(n67i(quTJ2H-G802MWx1{?ac@ z_$}4$)CQ8z93uMnQZ|#2@{Mkb*o?M$)S54Za62@|=e=8oDE~^ujrZ&%!Ip{X#%;;i z#u=}=_)cm{M4y)PqbcZPgYFm5Np$vZwXn z!a9;Wff~CO$6w<9o2I+GJC9dKuEgC+m9cfssUsx%T`9kha5hF*BA%r}(0>k7$Nz&F zdgDsWCU`Q%wSf&(w2(gc2k?q+WmxIY*!-r3%cZxw0 ztitvAgsk!)6f?%>Lwo&EyLPMJ8&TY8HqlKpJyt(!S?ynvxMd|-zs*+z*ItWo_M>7mI-P0^7b4)Y6 zQReLROmCFKByGi8rfcJe*wyPL8@;SG);9*Lb2Ov)8peIy;7JHz)%Bxjn1ZkTxCEo- zzGIX!rr5tnnN(~3o$L5T*~sT7%%O(eM3*AM42PYT0nprgoZ@6aPKpubOgY+{<_g}; zHiZ@wcOd&cTk*<$W510{xM5FYThP-pw(&liFUi6dMf=ye`hIQtz51@>*KH>~osl6e zZF}$0b2!7-Mwlkidd!YUrI6pYBJfwV%-WnVoiigldLlz;1n8u-u~ZEEoblv#gNLiS z!WSyKK>oz-I5&i`*VK(h)Y?Uxf%h@8zvQcWdbW(aRln-fePfMo+~A80zFhc5WX1dC z-t+kfo1Zd5?ZAEaE;h1TN3oN=XEEk=Kf=aevn6|WG3#tRerVcSGmF06qC3a8hvWvY zGUEJ(rQUDlf6EAbuZVo(M&C&X^<$9<6@m3x&`o!-a*5veb9mOT5MLt1IG7CbhHG{U z>_N+pd2i)c0sjprd=(UcL(>YQoHGkfl*rh3AV^OC>SUq{C^!cu<8O2P!zQC#9J*yY zOgHCor(0u-3H{-mM zQQd@1R+Xfo+Xva*@5~9bNf!pQ>J%ILesy$cqHxn`h}X_jX5b0ym)|HpOW5>xXvdXJ z`!Z^(gp=l`ex8QkGe9Ez&Ux_!F~6PHLO4{}Ia_Z%@446|4tjXE0CYDko4svbcF{+UyVy{?Ry5&c2<4jZ+*-o==bAQ$CHKu)68NGoAMiM z*ShYQQ-ph3E;NQT8emQt6O!znzW?~h^xx1TA=5p4xSUT?xtupolQTg&@TEfe3f)+C zQlXy-V}owPG(~v~My9&1O~x_0M=@JHIlMWqbvn3Gbncf{YXpt>C1+}yUo*}wFF~w8 zBlrAsWsUkU#?J)7s@w92->(WCV`5Jv#y2K&vI=uXrcBgiodQHwv;sDE7=pB+l+CV! zED+g^diF^=rP!N6cU?%z=^f9`1(QNQ;wF>r{ zBrTX6Ugh|4F$ITIk&HpiX%g-aSL9I?6>Sb1uX>ffuc=cRiu&*>hKrANJxK*>Nzk(Z zvEdtTm|A+zym{9V{BCOVV_)g49cfWG0`a-s?ZC}8LXMFVd}M{U>)(JR&9S1letCeK z3Opap;q=BZrHew+FAg?$!|Bbtva2?}E`!0Irk>75brfLM)fveaqZ(e3Z`tYeY<0R* zd@$nLMg!%1QIyC^yQkG;MALWA;7OOGpTf=$nW9O~YoE+r`68g2Mmcm{`$hvz4J*9= zN>I)9@U6#C1_56v^64PO3&>sQqV(2Im5`hz=7Bykm}bTz z%~Z2<3;lzGy*~e==c-^rKaAN2=QV{t?Z~qkSaIA}$&;`m9s_N5KJK z>T1$@Bki%{9~PvNg%qJmpo+O*p+Fxm>mWrWplV}j+%zzw62Qn~2w}=d)he`$skh#) zQ;a6^#KJ;o`CBI~ELoATg4yNrd2e^kVab3VD9p1s6)Y)>Jd^I6DXr%0fl^Tf6P$mn%*Ock$ue?$Es&iksE_{~k!E@cMe!PR9 zLKRAEYu838qBkZ>c(Nb+SvG#BZRd20GGV8AR!(UM?-LrUN>t)OYap98n1Ym)Yxr;d zH8VBe0s@O3ycZGedg|IG?3dB-47uTIchp(Q>v(@g3J83l%J;FyQ2OOsm7#(KhK(F5 zL@v2q_)02Lh(0)6u>p%HSejjN{btyJAO^LA`BI_K>(8JMw@1Ts!9P>O4XX_E@4MDz zeW@b{rxo+B<}#g-Np+0@+Vsc*ghAuve7xZhd!_cjmYi1@IO=r(Cl9`Kbm<)gTsjW- zT?wQBYs2m=Ii)^~8i&$Wo>qo7NCs(S?)brl;_{PRQPc6wAcN=4^jFzc=%mr}M@r3B zUhcjc_CpED;rccUp(wL3W6w32_Tf>_$_>UQ25|qc-hH~8u5ENd)&pEeB|5x+v{`H4 zU~SkHEhU@=Q{beLt?^3_dcjcz5XDA#Y&)47kdbP^PbtHQK=8ZU!fSf9wk@PLc zd(fS<%pb98DpSD|LkFm6;{MIGHek9U1R6ZaSKJ?P0GMUt1~X?WDo( zJFd86QO9WNxn_UtZHyR~`CMaWMo+7otdZ%~4IQl@NnlhVRF`x~SnB(L+_ulCY2XhQ z!c4Ru9f7zQaLn~)9jN-iuCg6!wcZKnpQN8bF}e<{pxkpn?H_;ZUg!)LMG{{jBO&j% zmco4rv9(*Sy_gv>Z;QCs=1*+Eds>W2DQw7kVs0 zb@)M3G2UC@wwPs3w!_(n=BmNH!ERlZTQ~kdKlwQs;lxIQ?{VZMYx^CH_ z@23{`);C=TX;1n_A4A*a>c19gEQ~zY0F5_)akE`&iTCiQsEDxxQ$U3HP#dYKE^`LX z(4UPhuG|m@Bo!+u?xQ};%r#-7FNChx6nw*=HPQ36v+_p@y4-hCHO62Uigj(cIGja> z?;}p8R}s!%&})+*0(y<#RKb}}d9##p3@#`RbpsRd$GZ*5^a3^QvMaZcBQ`RE}=% z9buWZ3!Ag?WF>I7mC;3=$p?G!SL3?`QTZ2o6QXY87_NF1YKwlWF=&N0a6Ez^_C;(;jmafo&Zn8^bSt)DmKdY&QGH-2Z?2-7)I+f6w<+dO_v$1DP46 z7LcOLv`m5*ybu5gOLCsZZ1d9&d^mu-zZ=&(9E*SA{_vkzHq5>&u24~I&FdHb{jJRN zfSTSk!T9W8q21M>uVoMUs~XOS!AjSGo&c-wxtZ4h;M+KT08pi@r5PQ}wSFV#C@)H2 z0PA%)*0!5I0NNIJL%1ACw9*Zc(<$TlHp#%YeQGReFS$ph9gd4tX(t_rN`;Dvo|Vho zQ;=&cnG4LgTxRA(0U&6${8?h^g*HW)?>jG^FuUhG-myp_2X(f##qAQ@>F;Lrtml}W z&cgf+0Kl*e$x3qDMxUL$2FdQPK*ow3H0MrG^yNfQLC7c51$7`7ftAf$e1Bg)yW-h7 zt0c-c0`-O*<${}EMj)r~nf1c|EcTqd7%YywSd!Io!4Zp(qQZCyrOyl{$_bRrQ_=OJ zW+`~Enlb?RN`zc__~rU%z3~UNzcW8ad^w= zX^-V28FjNF^J6j+m|WFxEhKERQ~Mm?Y1;gTE>0hp^aN1j^Qh!Jz?m#pxj?atST2F) zVIxoaO*(-u@ZQNIp<* zLU{J`JJ>N~gEGbN{i~2b1d>DU=)=NLp!Hs7armZtW%deHb*NU`h^Fjr9Giw{K-T`w zDRJ-g?Xr1ODP*C)Pgo$h1`b((VDzw_@Nl0m=5fK|{yQ>|4KIkfM6QQHvywAj9VloY z&Bg)akl)FEB!8QT&iiXt@d23i<_)o*Zs?{NT5PW! zjC?HUn77&xH(kE0p8E2n#_al`uH-(gQXjyS)&r98fC-+ll4NMsdKFL#ZZk0U;J9J) z{M?FUZy6x$3HhKaiCb~ZcWT+?_Scq!=7+M+=BdvE_#`14vt5c0JaJtyOzqW2u(UMq z4^jNnKiwy5I_LmV+lvW00daC6Vp|O*90L07Vp4&>R`E~ zeUGQL21{Ap;8mwj=FlqRbUfWjdwvi{EZ`}ctdf9lPbFxcc272(B8l52*|Y%1k=@a$p?5F^3^$d%8e+}6t8&}Eu^O*UK zW|gH>a!KRUspU&y>q=6)Lf1tA;v-qo@9~olD*2%3tc9ofFk{rDwo5f=AkLO`B)4cz zGuB~q)@unl=8NQQ5j^P>9y;j&I;^&aY@S$ZhT!{dau8bE4xM_Ye)SLdD%iSe?bqtq zRe!pa9^(xmb;ntHo*uj8&k+Oy)z0vILjhebt>_HbuQ$zV?Ust6;+bXpM|k%TQmrCL5?`~hsxi+VcVJg~a=xpfePE~r_#x&jdx_KV;%&<#K=l2j= z@=R|m4_Au=z4qO!_5+YP-`!%6%pn&wINA-gnk$82v)zibbCZ>r2nG~v=R zhVT*+g8ta&Qha*scCQQLw`P22xs6mGNhf(Kl1tA6aYj%06yUN3fZU8fDm70wTe;L- z?&yu@-|Q{mI<&6^@4Uv1tP4_uM|RASw@5ObS- zyunT0D=pVFN#z4>RC!A3JUZ@s_|2kA)IY;Zf*e@BHi3MxOS@$z-<|Himv+H^br zJWy|<`S4`)(qF{FF(Ed-6gt}}c*lJ@T~?v=d?hbq@#;ZQK;E0|ih#k@X5jkRoGkoG z*o5NHRC3@d$6x;`_Gj(@ALo8WhkAgqtB7JSqT#*ZKefMH88VY9A7n8KZ*@jc>R_5Y z1*uHW@g@`^gX@#T|EBCu1L@gs8{0WYryXpL=93j`Z35M6C&1A33#1chx*ZNM9%0Rq zNcnNv_l?$Ou?~H*@;k-UU*K;<`;_o!{K@p6W!56JmWPHo`Gb3lD9+W)yS6yKsa{@- z;No}rO#3&h{C7a@PtvpYHoB2JEXFjQ-mvGpppAr#zqJ|DLYs%z!eIBf0nabn`~xsN zjry}1hE~}AK^XDP|M6Sepd7NW81OI)3yv(nr2nrUd?Pb&wJ}wYTA-~dh4N9DK`~}0 z`R{nZy<(81%<+4GvB9*AyUah;Q!*3v|E&d*eIL)iSPq|cw4xPpLHg40^}o42nfiBL zPsHueKNbG>hvZ2N{;hBU0J+!x(hdOl=5v&R-I!ja@&rTE@G$Iol;;PIffgDj%te&>Lq4jkwZV97}8-DON8_DE4cTM{9 zSThsdx{%&@;EOJdPvw`)O6vb1_IhmE6|0k6uqAn(uGxX{=5?S>eRYS3P58t^jv~~x zNZUjFvN|xmK~&x15B@LAnaGY#NUY5CT2xQ?JlbZWNx5zi`B$bI{xIBFmNDN#;J+j> zWd0YQ1(0d?=hLtRH+KwWV#=Xl!tRzBcT6Zu#~FexuXcsN3BCJl{Oxi;XiuWMQSx#g zlqIe!H1K5yg&dV53c z;g+M)(5d9k$Z~g~pIw3B@mh?uwmR;>P%^l2RiXCy48(q2gM!y=@6T1j^+`J8W;J)85 zZc)|)_z8`=69i~iB8r^|Q)jA?T|4gI0y*b2=icUraMdp}6gW>2!^jAh#dqSpc&ntMFQ zqU&D-d2%y2xJ5dypw1(>vtp3|RXNCR#T}Y)tX~C5I&zFUPJo-Y5v=jOz!UI@(`VnM zgp|>vcQm%9c|N(T#Fl=e2U;2n6$``@r!wrkH^kN-5n7$HF|0mVnaFdkpqq=%Bm~hS zTR=|x4TXE-DPP1^x`+BpNLB~feSg8dp20qvwe3uwSlk<$vCc$Rc!P*PWmlhU!{PDr z#~(&3hKW4E(5Cb%VJ+k4r1)v0$w+g1eA4oikOx()P;x*BAq7VTQ)8`Ozn zTNNXdu7U;kL!BXy*@P@zAP%F*(ia6dyP_gLqhaHVEZ`VVvV(~E;~@ShbM8*vXb^L0 zO$OB72N9jf^CdrtolInXMzB!aZ|9?+iJi1J>;Ja2qoC{1&FErO~n-%>Mjh{2{< zW&!sH@yrAmQ`X5#2z~Uj4Sq>!h7ljrDxPo>(w)w+36FjGuUm7@xO7>#j0?(elZ{ER zxs19SqI!9k>N;Dk^%%6J`yz##uxm6k;kKNVVElkaRDPfDP6of3OQhG73)vv#H`#4K z`ZAHuwb85?Qv^MNz^X2j^Hje3s;+u3 zTB2`fq=$BKj#YgwVDBiXRkH!P8GQC4!NYXRt)5_f0UF~^+ zze80(iP{j0U#`r_y+i-S^tX&tR>)bA?diu|V3B^0!DgVK!D&X)UeSZT!EdRuF2quJ z%Qw2%n>5R3KMVzm)smL0XFRg{;WaDoOuHYkH(dOF4MpzCYkFWUsXM1mq-uLG7M}}k z6{@fq%i^tPB!KNg3=usMrJP<>pdo&>G+%<205)W=rQMPz1Ty||SE|>I4m7*bx#Q9~ znYDg#0+9SiXvxNI%pV`p)14^`EE-Z$H^mx}-3)>DnRAAjt%=MH6QSE?+;d*yqUOI) z?uCw8#b-;JBtGdF42Su3UD0TiD8>A6^1a4m_OVx|FHHGuVur8LoK&%p&DbMT?#~H_ z*Gfz`sY<{`<;M^+1Q!SNL%j?GH!Qr=;92P=8g&I(DBNk27(Yz|HsLm~u)ApdI-2NK zZ-Cdrf+!K?=MnA`&n2a%i3)e$9oZl4Ej_k1e2l5($G+gwazS(5+V+O`>I zDD7l7{c5RHD*tAr4g*~-f`}Y!P5z*n=1C$ZlK37grd>6zf26|Sklh6zDgRCdKsn>=}y?t3X{ zZhYrByIX6OJGj;5MW~}~(br@`ZwSEnS*&W=Ha<6z$H(*}_C|@08maSIO~GiZIyC6e zeHMt-qq6AxaojfQ*;7Ai{t{<|>~?fzl$(#W%b!JbvL@!Sfh%b4BdW_=lDzG{;vJ=usdE zAtP9&{b>9?tbm$iACGdE*v&is+G=^S%FYMRZ^qdPnB2<5Zlw*8@r*`OyqL`1f#y64 z-L@KS_5EmW*Jr+s)K-Jv?&Aj*jtQjSt~B{oiy}dT(2HI<*xePIIEu81D9hu{v=YvQ zh!sLiTl+y!Ud9z&(D)xKrCypiCtj5+iRS^&dRH+!gx{Mm&~EP-QnZ7raCr&VXFS2yKI*ak>_nJBIDAHUuRNdT1y3E ztB}L&A3G4$GF>%y6MVLfklZ`8V}cWtxg)jV++tDfcl~RI0)#pGF+XeniuY=U7d6^g zypxROQxSEEy&$nlkz`g#?2zpJu@6y&%%f9w1T_wI1vw*o48FJW7k%J*wUJSP?gyV3 zW^Oh}l*Jdvs2wb~%=~78SmHi-7ygCsycJY6>1@eHQ$@mS&XiY$71$oCn7+xBrHlrG z7GU^PLf!e^7GO0qLf__L+pL!Aou;b+Dsq zI$_V5J&rxwx{O{u>1GQ&el31F9w%&pyH<2RCFIn?V_>KF%zCYrr2=eG>DR5P82qcY zYF|db&+>$bb)5?Rty48Zk~@6Ia=2QvTYY-8i6LxWwFpH^g>o649I2lKy>$J3kxvkn zA&|9KGO<(3>hX%SbQE1`18X<>DX*P51`ZIQ)^vJ^2sEAnQ zrFiGZ7}W&g`!Oo>%#+((xjiXo7E`te`WN|4Q=>%#2@( zg&V>_z1(`+P8QHb&dTl}Y7ux(m<*yyWg5xz#vw?wJ^Q(Y@E7U~pn zO@V3J$;;<>fxYptm>)+#71mFOHlLgyoBJT0&2y;MRvdbUS`ui-rBnMqR2r(51-{MA15)pp7C%PGk%>y#(;D<1zF7dXG)Fj6#-JEXY zl|(6Q!Tr_hlhw0?>g**#{0@m=d3H0FF8MB>E{L|72NQMor^qW_%Pm)EySeS**Jn^( z0u%ipLUfeqZ%8>T`xgQw#rgCyzBzYU zTUnik7FLyu+U&vzVY5v53N{*{(piJ&uGzpa}vZB(ks>zs<)%+z)MQiPrS@hwVYz z_Mh4!h%Z-AmC>P-!ETzkwqm>0k9Syh@HuYu?Juxp}yn&$jE$z30aW5vw!#tUqe=l4@LHy05ifty~NOju#aY%p35z7r~J zD9{_9$X-Cm-$A*urRbs$4$GZB3&~j!Qpc;PdwY)^luDp;-A?3O!y`Rl8|OQU7aolX z5bVwK8nZe}Gd=g!v=9^M44icjA(C(&mcd*mOJGttHIqTvl$^v@tdK0$tJC5T{%Zr2Yq?D!^df7n~1m>=mB>t zn-2jb;pS-AOW~(i-Zaezd>tQ%i95`-?h)pTZ5@gD*!Z(Iguui2a9VVG`17T*XHBmL zXY68FxQ*5PVO_0tN_aN