From c5a1ba925cea60a6c5cacfdade15644a7028eab0 Mon Sep 17 00:00:00 2001 From: mdbirnstiehl Date: Fri, 2 Aug 2024 16:42:16 -0500 Subject: [PATCH] review updates --- README.md | 23 ++++++++++++----------- docs/collector-components.md | 9 ++++----- docs/collector-limitations.md | 6 +++--- docs/configure-upstream-collector.md | 9 +++------ docs/guided-onboarding.md | 16 ++++++++-------- docs/manual-configuration.md | 12 ++++++------ 6 files changed, 36 insertions(+), 39 deletions(-) diff --git a/README.md b/README.md index 3debd30..9b48d7c 100644 --- a/README.md +++ b/README.md @@ -1,26 +1,27 @@ -# Elastic OpenTelemetry Distributions +# Elastic Distributions for OpenTelemetry [OpenTelemetry](https://opentelemetry.io/docs/) is a vendor- and tool-neutral observability framework for collecting, processing, and exporting telemetry data. +The Elastic Distribution for OpenTelemetry (Elastic OTel Collector) is a supported distribution of the [OpenTelemetry Collector](https://github.com/open-telemetry/opentelemetry-collector) made up of mostly upstream components. You can send your telemetry data to Elastic Observability using OpenTelemetry the following ways: -- Use the **Elastic Distribution for OpenTelemetry Collector**, an extension of the [OpenTelemetry Collector](https://github.com/open-telemetry/opentelemetry-collector), to collect and send logs and host metrics to [Elastic Cloud](https://cloud.elastic.co/). -- Use the **Elastic Distributions for OpenTelemetry**, extensions of select [OpenTelemetry language SDKs](https://opentelemetry.io/docs/languages/), to instrument your applications and send logs, traces, and metrics to [Elastic Cloud](https://cloud.elastic.co/). The following languages are currently supported: Java, .NET, Node.js, and Python. -- Configure your upstream [custom collector](https://opentelemetry.io/docs/collector/custom-collector/) or [contrib](https://github.com/open-telemetry/opentelemetry-collector-contrib) distribution for the collector to collect logs and metrics and send them to Elastic Observability. +- Collect and send logs and host metrics to [Elastic Cloud](https://cloud.elastic.co/) using the Elastic OTel Collector. +- Instrument your applications and send logs, traces, and metrics to [Elastic Cloud](https://cloud.elastic.co/) using the **Elastic Distributions for OpenTelemetry** extensions of select [OpenTelemetry language SDKs](https://opentelemetry.io/docs/languages/). The following languages are currently supported: Java, .NET, Node.js, and Python. +- Configure an upstream [custom collector](https://opentelemetry.io/docs/collector/custom-collector/) or [contrib](https://github.com/open-telemetry/opentelemetry-collector-contrib) distribution for the collector to collect logs and metrics and send them to Elastic Observability. This diagram provides a quick overview on how the different components work together. Refer to the [components](docs/collector-components.md) for a more in-depth look. ![Diagram of the OpenTelemetry flow](docs/images/elastic-otel-overview.png) -## Get started with the Elastic Distribution for OpenTelemetry Collector +## Collect infrastructure data with the Elastic Distribution for OpenTelemetry Collector -These pages detail the components and how to configure the Elastic Distribution for OpenTelemetry Collector ("the collector"). +These pages detail the components and how to configure the Elastic OTel Collector. - [Components](docs/collector-components.md): Get details on the components used to receive, process, and export telemetry data. -- [Guided onboarding](docs/guided-onboarding.md): Use the guided onboarding in Kibana or in a serverless Observability project to send data using the collector. -- [Manual configurations](docs/manual-configuration.md): Manually configure the collector to send data to Elastic Observability. -- [Limitations](docs/collector-limitations.md): Understand the current limitations of the collector. +- [Guided onboarding](docs/guided-onboarding.md): Use the guided onboarding in Kibana or in a serverless Observability project to send data using the Elastic OTel Collector. +- [Manual configurations](docs/manual-configuration.md): Manually configure the Elastic OTel Collector to send data to Elastic Observability. +- [Limitations](docs/collector-limitations.md): Understand the current limitations of the Elastic OTel Collector. -## Collect application data with Elastic Distributions for OpenTelemetry +## Collect application data with Elastic Distributions for OpenTelemetry language SDKs Elastic offers several distributions that extend [OpenTelemetry language SDKs](https://opentelemetry.io/docs/languages/). The following languages are currently supported: @@ -31,4 +32,4 @@ Elastic offers several distributions that extend [OpenTelemetry language SDKs](h ## Configure an upstream collector for Elastic -[Configure a custom collector or a contrib distribution of the collector](docs/configure-upstream-collector.md): Configure a [custom collector](https://opentelemetry.io/docs/collector/custom-collector/) or [contrib](https://github.com/open-telemetry/opentelemetry-collector-contrib) distribution for the collector to collect logs and metrics and send them to Elastic Observability. \ No newline at end of file +[Configure a custom collector or a contrib distribution of the OpenTelemetry collector](docs/configure-upstream-collector.md): Configure a [custom collector](https://opentelemetry.io/docs/collector/custom-collector/) or [contrib](https://github.com/open-telemetry/opentelemetry-collector-contrib) distribution for the OpenTelemetry collector to collect logs and metrics and send them to Elastic Observability. \ No newline at end of file diff --git a/docs/collector-components.md b/docs/collector-components.md index dc097a8..15e6656 100644 --- a/docs/collector-components.md +++ b/docs/collector-components.md @@ -1,4 +1,4 @@ -# Elastic OpenTelemetry Collector components +# Elastic Distribution for OpenTelemetry Collector components The OpenTelemetry Collector uses the following components to receive, process, and export telemetry data: @@ -7,7 +7,7 @@ The OpenTelemetry Collector uses the following components to receive, process, a - [Exporters](collector-components.md#exporters): Send data to the backends or destinations. - [Extensions](collector-components.md#extensions): Provide additional functionalities and capabilities. -The default configurations of the Elastic Distribution for the OpenTelemetry Collector follows these flows. +The default configurations of the Elastic OTel Collector follows these flows. **MacOS and Linux Host metrics:** @@ -65,7 +65,7 @@ flowchart LR style eight fill:#e6f9f7,stroke:#333,stroke-width:1px ``` -**Kubernetes logs** +**Kubernetes, MacOS, and Linux logs** ```mermaid flowchart LR one["`File log receiver`"] @@ -84,7 +84,7 @@ flowchart LR style six fill: #e6f9f7,stroke:#333,stroke-width:1px ``` -Refer to the following tables for more information on the components supported by the Elastic Distribution for the OpenTelemetry Collector. +Refer to the following tables for more information on the components included in the Elastic Distribution for the OpenTelemetry Collector. Follow the links for OpenTelemetry documentation with more configuration details for each component. To set up the collector, go to the [guided onboarding](guided-onboarding.md) or [manual configuration](manual-configuration.md) to get started. @@ -104,7 +104,6 @@ To set up the collector, go to the [guided onboarding](guided-onboarding.md) or | Component | Description | |---|---| -| [`elasticinframetricsprocessor`](https://github.com/elastic/opentelemetry-collector-components/blob/processor/elasticinframetricsprocessor/v0.7.1/processor/elasticinframetricsprocessor/README.md) | Processes infrastructure metrics to enhance and convert them for Elasticsearch. | | [`attributesprocessor`](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/processor/attributesprocessor/v0.105.0/processor/attributesprocessor/README.md) | Modifies telemetry data attributes. | | [`filterprocessor`](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/processor/filterprocessor/v0.105.0/processor/filterprocessor/README.md) | Filters telemetry data to include or exclude specific data points. | | [`k8sattributesprocessor`](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/processor/k8sattributesprocessor/v0.105.0/processor/k8sattributesprocessor/README.md) | Enhances telemetry data with Kubernetes-specific metadata. | diff --git a/docs/collector-limitations.md b/docs/collector-limitations.md index 683c428..ec62720 100644 --- a/docs/collector-limitations.md +++ b/docs/collector-limitations.md @@ -1,8 +1,8 @@ -# Elastic OpenTelemetry Collector limitations +# Elastic Distribution for OpenTelemetry Collector limitations -The Elastic Distribution for the OpenTelemetry Collector has the following limitations: +The Elastic OTel Collector has the following limitations: -- `host.network.*` metrics aren't present from the OpenTelemetry side. +- Because of an upstream limitation, `host.network.*` metrics aren't present from the OpenTelemetry side. - `process.state` isn't present in the OpenTelemetry host metric. It's set to a dummy value of **Unknown** in the **State** column of the host processes table. - The Elasticsearch exporter handles the metadata fields, but **Host OS version** and **Operating system** may show as "N/A" and **Host IP** may show different values. - The CPU scraper needs to be enabled to collect the `systm.load.cores` metric, which affects the **Normalized Load** column in the **Hosts** table and the **Normalized Load** visualization on the host detailed view. \ No newline at end of file diff --git a/docs/configure-upstream-collector.md b/docs/configure-upstream-collector.md index a9405be..6ba3234 100644 --- a/docs/configure-upstream-collector.md +++ b/docs/configure-upstream-collector.md @@ -2,12 +2,9 @@ You can configure an upstream collector, like a [custom collector](https://opentelemetry.io/docs/collector/custom-collector/) or [contrib](https://github.com/open-telemetry/opentelemetry-collector-contrib) distribution for the collector, to collect logs and metrics and send them to Elastic Observability. - - For a more seamless experience, use the Elastic distribution for the OpenTelemetry collector. Refer to the [guided onboarding](guided-onboarding.md) docs or the [manual configuration](manual-configuration.md) docs for more on configuring the Elastic distribution. -## Upstream collector configuration example -Use the following configuration example as a reference when configuring an upstream collector. - - \ No newline at end of file +## Upstream collector configuration examples +Refer to the OpenTelemetry documentation on [building a custom collector](https://opentelemetry.io/docs/collector/custom-collector/) for more on creating an upstream collector. +Use the Elastic [example configurations](https://github.com/elastic/elastic-agent/tree/main/internal/pkg/otel/samples) as a reference when configuring your upstream collector. \ No newline at end of file diff --git a/docs/guided-onboarding.md b/docs/guided-onboarding.md index 6ec7f58..240cc9f 100644 --- a/docs/guided-onboarding.md +++ b/docs/guided-onboarding.md @@ -1,18 +1,18 @@ # Collect logs and metrics using the guided onboarding -The guided onboarding in Kibana or in a serverless Observability project walks you through collecting logs and metrics using the Elastic OpenTelemetry Collector. -If you prefer manually configuring the Elastic OpenTelemetry collector, refer to [Manually configure the collector](manual-configuration.md). +The guided onboarding in Kibana or in a serverless Observability project walks you through collecting logs and metrics using the Elastic OTel Collector. +For manual configuration, refer to [Manually configure the Elastic OTel Collector](manual-configuration.md). ## Before you begin The onboarding has the following requirements and limitations: - The **Admin** role or higher is required to onboard system logs and metrics. To learn more, refer to [Assign user roles and privileges](https://www.elastic.co/docs/current/serverless/general/assign-user-roles). - Root privileges on the host are required to run the OpenTelemetry collector used in this quickstart. -- The collector only works on Kubernetes, Linux, and MacOS systems. -- Refer to [Elastic OpenTelemetry Collector limitations](collector-limitations.md) for known limitations when using Elastic Distribution for the OpenTelemetry collector. +- The Elastic OTel Collector only works on Kubernetes, Linux, and MacOS systems. +- Refer to [Elastic OpenTelemetry Collector limitations](collector-limitations.md) for known limitations when using the Elastic OTel Collector. ## Collect your logs and metrics -Follow these steps to collect logs and metrics using the Elastic OpenTelemetry collector +Follow these steps to collect logs and metrics using the Elastic OTel collector 1. Open an [Elastic Cloud](cloud.elastic.co) deployment or a serverless Observability project. 1. To open the guided onboarding, either: @@ -23,10 +23,10 @@ Follow these steps to collect logs and metrics using the Elastic OpenTelemetry c 1. For **MacOS and Linux**, copy the command, open a terminal on your host, and run the command to download and configure the OpenTelemetry collector. 1. For **Kubernetes**, download the manifest. 1. Copy the command under Step 2: - 1. For **MacOS and Linux**, run the command in your terminal to start the OpenTelemetry collector. - 1. For **Kubernetes**, run the command from the directory where you downloaded the manifest to install the collector on every node of your cluster. + 1. For **MacOS and Linux**, run the command in your terminal to start the Elastic OTel collector. + 1. For **Kubernetes**, run the command from the directory where you downloaded the manifest to install the Elastic OTel Collector on every node of your cluster. -Logs are collected from setup onward, so you won't see logs that occurred before starting the collector. +Logs are collected from setup onward, so you won't see logs that occurred before starting the Elastic OTel Collector. The default log path is `/var/log/*`. To update the path, modify `otel.yml`. Under **Visualize your data**, you'll see links to **Logs Explorer** to view your logs and **Hosts** to view your host metrics. \ No newline at end of file diff --git a/docs/manual-configuration.md b/docs/manual-configuration.md index 25c420e..3eb78d7 100644 --- a/docs/manual-configuration.md +++ b/docs/manual-configuration.md @@ -1,24 +1,24 @@ -# Manually configure the collector -Collecting logs and host metrics with the Elastic Distribution for the OpenTelemetry Collector without using the [guided onboarding](guided-onboarding.md) requires some manual configuration. +# Manually configure the Elastic Distribution for OpenTelemetry Collector +Collecting logs and host metrics with the Elastic OTel Collector without using the [guided onboarding](guided-onboarding.md) requires some manual configuration. ## Before you begin -The onboarding has the following requirements and limitations: +The Elastic OTel collector has the following requirements and limitations: - The **Admin** role or higher is required to onboard system logs and metrics. To learn more, refer to [Assign user roles and privileges](https://www.elastic.co/docs/current/serverless/general/assign-user-roles). - Root privileges on the host are required to run the OpenTelemetry collector used in this quickstart. -- The collector only works on Kubernetes, Linux, and MacOS systems. +- The Elastic OTel Collector only works on Kubernetes, Linux, and MacOS systems. - Refer to [Elastic OpenTelemetry Collector limitations](collector-limitations.md) for known limitations when using the Elastic Distribution for the OpenTelemetry collector. ## Collect your logs and metrics -To manually configure the collector, gather the following information: +To manually configure the Elastic OTel Collector, gather the following information: - **Your Elasticsearch endpoint**: From the help menu in Kibana or your serverless Observability project, select **Connection details** and copy the **Elasticsearch endpoint**. - **API key**: - **Kibana:** From the help menu, select **Connection details** and select **Create and manage API keys**. From the **API keys** page, select **Create API key**. Give your API key a name, select **Create API key**, and copy the new API key. - **Serverless:** From the help menu, select **Connection details** and select the **API key** tab. Give your API key a name, select **Create API key**, and copy the new API key. -Then manually configure the Elastic OpenTelemetry Collector to collect logs and metrics on a MacOS or Linux system: +Then manually configure the Elastic OTel Collector to collect logs and metrics on a MacOS or Linux system: 1. Download and extract the standalone Elastic Agent for your platform. For more on downloading and extracting a standalone Elastic Agent, refer to the first step in [Install standalone Elastic Agents](https://www.elastic.co/guide/en/fleet/current/install-standalone-elastic-agent.html). 1. From the Elastic Agent base directory, go to the `otel_samples` directory. The `platformlogs_hostmetrics.yml` file has the configurations for the receivers, processors, and exporters needed to collect logs and host metrics.