diff --git a/README.md b/README.md index 4cf1944..d101e33 100644 --- a/README.md +++ b/README.md @@ -1,2 +1,35 @@ -# opentelemetry -Get started with our Elastic Distros of OpenTelemetry +# Elastic Distributions for OpenTelemetry + +[OpenTelemetry](https://opentelemetry.io/docs/) is a vendor-neutral observability framework for collecting, processing, and exporting telemetry data. +The Elastic Distribution for OpenTelemetry Collector (Elastic OTel Collector) is a supported, drop-in replacement distribution of the [OpenTelemetry Collector](https://github.com/open-telemetry/opentelemetry-collector) made up of mostly upstream components. +You can send your telemetry data to Elastic Observability using OpenTelemetry the following ways: + +- Collect and send logs and host metrics to [Elastic Cloud](https://cloud.elastic.co/) using the Elastic OTel Collector. +- Instrument your applications and send logs, traces, and metrics to [Elastic Cloud](https://cloud.elastic.co/) using the Elastic Distributions for select [OpenTelemetry language SDKs](https://opentelemetry.io/docs/languages/). Currently, Elastic provides distributions for the following language SDKs: Java, .NET, Node.js, and Python. +- Build and configure a [custom collector](https://opentelemetry.io/docs/collector/custom-collector/) or extend the [OpenTelemetry Collector Contrib](https://github.com/open-telemetry/opentelemetry-collector-contrib) distribution to collect logs and metrics and send them to Elastic Observability. + +This diagram provides a quick overview on how the different components work together. Refer to the [components](docs/collector-components.md) for a more in-depth look. + +![Diagram of the OpenTelemetry flow](docs/images/elastic-otel-overview.png) + +## Collect infrastructure data with the Elastic Distribution for OpenTelemetry Collector + +These pages detail the components and how to configure the Elastic OTel Collector. + +- [Components](docs/collector-components.md): Get details on the components used to receive, process, and export telemetry data. +- [Guided onboarding](docs/guided-onboarding.md): Use the guided onboarding in Kibana or in a serverless Observability project to send data using the Elastic OTel Collector. +- [Manual configurations](docs/manual-configuration.md): Manually configure the Elastic OTel Collector to send data to Elastic Observability. +- [Limitations](docs/collector-limitations.md): Understand the current limitations of the Elastic OTel Collector. + +## Collect application data with Elastic Distributions for OpenTelemetry language SDKs + +Elastic offers several Distributions that extend [OpenTelemetry language SDKs](https://opentelemetry.io/docs/languages/). The following languages are currently supported: + +* [Java](https://github.com/elastic/elastic-otel-java) +* [.NET](https://github.com/elastic/elastic-otel-dotnet) +* [Node.js](https://github.com/elastic/elastic-otel-node) +* [Python](https://github.com/elastic/elastic-otel-python) + +## Configure a custom collector or the OpenTelemetry Collector Contrib distribution + +[Configure a custom collector or the OpenTelemetry Collector Contrib distribution](docs/configure-upstream-collector.md): Build and configure a [custom collector](https://opentelemetry.io/docs/collector/custom-collector/) or extend the [OpenTelemetry Collector Contrib](https://github.com/open-telemetry/opentelemetry-collector-contrib) distribution to collect logs and metrics and send them to Elastic Observability. \ No newline at end of file diff --git a/docs/collector-components.md b/docs/collector-components.md new file mode 100644 index 0000000..0c26589 --- /dev/null +++ b/docs/collector-components.md @@ -0,0 +1,131 @@ +# Elastic Distribution for OpenTelemetry Collector components + +The OpenTelemetry Collector uses the following components to receive, process, and export telemetry data: + +- [Receivers](collector-components.md#receivers): Collect telemetry from your host. +- [Processors](collector-components.md#processors): Modify or transform telemetry data before sending it to the exporters. +- [Exporters](collector-components.md#exporters): Send data to the backends or destinations. +- [Extensions](collector-components.md#extensions): Provide additional functionalities and capabilities. + +The default configurations of the Elastic Distribution for the OpenTelemetry Collector follows these flows. + +**MacOS and Linux Host metrics:** + +```mermaid +flowchart LR + one["`Host metrics receiver`"] + two["`Elastic infra metrics processor`"] + three["`Attributes processor (dataset)`"] + four["`Resource processor (process)`"] + five["`Elasticsearch exporter`"] + one --> two --> three --> four --> five + + style one fill: #e6f9f7,stroke:#333,stroke-width:1px + style two fill: ##f8e9e9,stroke:#333,stroke-width:1px + style three fill: ##f8e9e9,stroke:#333,stroke-width:1px + style four fill: ##f8e9e9,stroke:#333,stroke-width:1px + style five fill:#e6f9f7,stroke:#333,stroke-width:1px +``` + +**MacOS and Linux Logs** + +```mermaid +flowchart LR + one["`File log receiver`"] + two["`Resource detection processor`"] + three["`Elasticsearch exporter`"] + one --> two --> three + + style one fill: #e6f9f7,stroke:#333,stroke-width:1px + style two fill: ##f8e9e9,stroke:#333,stroke-width:1px + style three fill:#e6f9f7,stroke:#333,stroke-width:1px +``` + +**Kubernetes metrics** + +```mermaid +flowchart LR + one["`Kubelet stats and host metrics receivers`"] + two["`Kubernetes attributes processor`"] + three["`Elastic infra metrics processor`"] + four["`Resource detection processors (EKS, GCP, K8s)`"] + five["`Resource processors (K8s, cloud)`"] + six["`Attributes processor (dataset)`"] + seven["`Resource processor`"] + eight["`Elasticsearch exporter`"] + one --> two --> three --> four --> five --> six --> seven --> eight + + style one fill: #e6f9f7,stroke:#333,stroke-width:1px + style two fill: ##f8e9e9,stroke:#333,stroke-width:1px + style three fill: ##f8e9e9,stroke:#333,stroke-width:1px + style four fill: ##f8e9e9,stroke:#333,stroke-width:1px + style five fill: ##f8e9e9,stroke:#333,stroke-width:1px + style six fill: ##f8e9e9,stroke:#333,stroke-width:1px + style seven fill: ##f8e9e9,stroke:#333,stroke-width:1px + style eight fill:#e6f9f7,stroke:#333,stroke-width:1px +``` + +**Kubernetes, MacOS, and Linux logs** +```mermaid +flowchart LR + one["`File log receiver`"] + two["`Kubernetes attributes processor`"] + three["`Resource detection processors (system, EKS, GCP)`"] + four["`Resource processors (K8s, cloud)`"] + five["`Attributes processor (k8s_logs_dataset)`"] + six["`Elasticsearch and debug exporters`"] + one --> two --> three --> four --> five --> six + + style one fill: #e6f9f7,stroke:#333,stroke-width:1px + style two fill: ##f8e9e9,stroke:#333,stroke-width:1px + style three fill: ##f8e9e9,stroke:#333,stroke-width:1px + style four fill: ##f8e9e9,stroke:#333,stroke-width:1px + style five fill: ##f8e9e9,stroke:#333,stroke-width:1px + style six fill: #e6f9f7,stroke:#333,stroke-width:1px +``` + +Refer to the following tables for more information on the components included in the Elastic OTel Collector. +Follow the links for OpenTelemetry documentation with more configuration details for each component. +To set up the Elastic OTel Collector, get started using the [guided onboarding](guided-onboarding.md) docs or the [manual configuration](manual-configuration.md) docs. + +## Receivers + +| Component | Description | +|---|---| +| [`filelogreceiver`](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/receiver/filelogreceiver/v0.105.0/receiver/filelogreceiver/README.md) | Collects logs from files on the local filesystem, supporting various formats and log rotation strategies. | +| [`hostmetricsreceiver`](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/receiver/hostmetricsreceiver/v0.105.0/receiver/hostmetricsreceiver/README.md) | Collects metrics from the host machine, such as CPU, memory, disk, and network usage. | +| [`httpcheckreceiver`](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/receiver/httpcheckreceiver/v0.105.0/receiver/httpcheckreceiver/README.md) | Performs HTTP checks to monitor the availability and response time of web services. | +| [`k8sclusterreceiver`](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/receiver/k8sclusterreceiver/v0.105.0/receiver/k8sclusterreceiver/README.md) | Gathers metrics and metadata from a Kubernetes cluster. | +| [`k8sobjectsreceiver`](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/receiver/k8sobjectsreceiver/v0.105.0/receiver/k8sobjectsreceiver/README.md) | Monitors changes to Kubernetes objects, and collects related metrics. | +| [`kubeletstatsreceiver`](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/receiver/kubeletstatsreceiver/v0.105.0/receiver/kubeletstatsreceiver/README.md) | Collects metrics from the Kubelet, including node and pod-level resource usage. | +| [`otlpreceiver`](https://github.com/open-telemetry/opentelemetry-collector/blob/receiver/otlpreceiver/v0.105.0/receiver/otlpreceiver/README.md) | Receives metrics, traces, and logs in OpenTelemetry Protocol (OTLP) format. | + +## Processors + +| Component | Description | +|---|---| +| [`elasticinframetricsprocessor`](https://github.com/elastic/opentelemetry-collector-components/blob/processor/elasticinframetricsprocessor/v0.7.1/processor/elasticinframetricsprocessor/README.md) | Processes infrastructure metrics to enhance and convert them for Elasticsearch. | +| [`attributesprocessor`](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/processor/attributesprocessor/v0.105.0/processor/attributesprocessor/README.md) | Modifies telemetry data attributes. | +| [`filterprocessor`](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/processor/filterprocessor/v0.105.0/processor/filterprocessor/README.md) | Filters telemetry data to include or exclude specific data points. | +| [`k8sattributesprocessor`](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/processor/k8sattributesprocessor/v0.105.0/processor/k8sattributesprocessor/README.md) | Enhances telemetry data with Kubernetes-specific metadata. | +| [`resourcedetectionprocessor`](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/processor/resourcedetectionprocessor/v0.105.0/processor/resourcedetectionprocessor/README.md) | Detects resource attributes and adds them to telemetry data. | +| [`resourceprocessor`](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/processor/resourceprocessor/v0.105.0/processor/resourceprocessor/README.md) | Allows resource attributes to be modified in telemetry data. | +| [`transformprocessor`](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/processor/transformprocessor/v0.105.0/processor/transformprocessor/README.md) | Transforms telemetry data modifying based on specified rules. | +| [`batchprocessor`](https://github.com/open-telemetry/opentelemetry-collector/blob/processor/batchprocessor/v0.105.0/processor/batchprocessor/README.md) | Batches telemetry data to improve export performance and manage load on back-end systems. | + +## Exporters + +| Component | Description | +|---|---| +| [`elasticsearchexporter`](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/exporter/elasticsearchexporter/v0.105.0/exporter/elasticsearchexporter/README.md) | Sends collected telemetry data to Elasticsearch for storage and analysis. | +| [`fileexporter`](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/exporter/fileexporter/v0.105.0/exporter/fileexporter/README.md) | Writes telemetry data to a file, useful for debugging or offline analysis. | +| [`debugexporter`](https://github.com/open-telemetry/opentelemetry-collector/blob/exporter/debugexporter/v0.105.0/exporter/debugexporter/README.md) | Outputs telemetry data in a human-readable format for debugging purposes. | +| [`otlpexporter`](https://github.com/open-telemetry/opentelemetry-collector/blob/exporter/otlpexporter/v0.105.0/exporter/otlpexporter/README.md) | Sends telemetry data in OTLP format to a specified endpoint. | +| [`otlphttpexporter`](https://github.com/open-telemetry/opentelemetry-collector/blob/exporter/otlphttpexporter/v0.105.0/exporter/otlphttpexporter/README.md) | Sends telemetry data using HTTP with OTLP. | + +## Extensions + +| Component | Description | +|---|---| +| [`filestorage`](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/extension/storage/filestorage/v0.105.0/extension/storage/filestorage/README.md)| Provides file-based storage for temporary data, such as checkpoints and intermediate state. | +| [`memorylimiterextension`](https://github.com/open-telemetry/opentelemetry-collector/blob/extension/memorylimiterextension/v0.105.0/extension/memorylimiterextension/README.md) | Limits the memory usage of the collector to prevent out-of-memory errors. | \ No newline at end of file diff --git a/docs/collector-limitations.md b/docs/collector-limitations.md new file mode 100644 index 0000000..abfe3cd --- /dev/null +++ b/docs/collector-limitations.md @@ -0,0 +1,11 @@ +# Elastic Distribution for OpenTelemetry Collector limitations + +The Elastic Distribution for the OpenTelemetry Collector has the following limitations: + +- Because of an upstream limitation, `host.network.*` metrics aren't present from the OpenTelemetry side. +- `process.state` isn't present in the OpenTelemetry host metric. It's set to a dummy value of **Unknown** in the **State** column of the host processes table. +- The Elasticsearch exporter handles the resource attributes, but **Host OS version** and **Operating system** may show as "N/A". +- The CPU scraper needs to be enabled to collect the `systm.load.cores` metric, which affects the **Normalized Load** column in the **Hosts** table and the **Normalized Load** visualization on the host detailed view. +- The [`hostmetrics receiver`](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/hostmetricsreceiver) doesn't support CPU and disk metrics on MacOS. These values will stay empty for collectors running on MacOS. +- The console shows error Log messages when the [`hostmetrics receiver`](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/hostmetricsreceiver) can't access some of the process information due to permission issues. +- The console shows mapping errors initially until mapping occurs. \ No newline at end of file diff --git a/docs/configure-upstream-collector.md b/docs/configure-upstream-collector.md new file mode 100644 index 0000000..2a76ca3 --- /dev/null +++ b/docs/configure-upstream-collector.md @@ -0,0 +1,10 @@ +# Configure a custom collector or the OpenTelemetry Collector Contrib distribution + +You can build and configure a [custom collector](https://opentelemetry.io/docs/collector/custom-collector/) or extend the [OpenTelemetry Collector Contrib ](https://github.com/open-telemetry/opentelemetry-collector-contrib) distribution to collect logs and metrics and send them to Elastic Observability. + +For a more seamless experience, use the Elastic Distribution for the OpenTelemetry collector. +Refer to the [guided onboarding](guided-onboarding.md) docs or the [manual configuration](manual-configuration.md) docs for more on configuring the Elastic OTel collector. + +## Upstream collector configuration examples +Refer to the OpenTelemetry documentation on [building a custom collector](https://opentelemetry.io/docs/collector/custom-collector/) for more on creating an upstream collector. +Use the Elastic [example configurations](https://github.com/elastic/elastic-agent/tree/main/internal/pkg/otel/samples) as a reference when configuring your upstream collector. \ No newline at end of file diff --git a/docs/guided-onboarding.md b/docs/guided-onboarding.md new file mode 100644 index 0000000..491c148 --- /dev/null +++ b/docs/guided-onboarding.md @@ -0,0 +1,32 @@ +# Collect logs and metrics using the guided onboarding +The guided onboarding in Kibana or in a serverless Observability project walks you through collecting logs and metrics using the Elastic Distribution for OpenTelemetry Collector. +To configure the Elastic OTel Collector manually, refer to the [manual configuration](manual-configuration.md) docs. + +## Before you begin +The onboarding has the following requirements and limitations: + +- The **Admin** role or higher is required to onboard system logs and metrics. To learn more, refer to [Assign user roles and privileges](https://www.elastic.co/docs/current/serverless/general/assign-user-roles). +- Root privileges on the host are required to run the OpenTelemetry collector used in this quickstart. +- The Elastic OTel Collector only works on Kubernetes, Linux, and MacOS systems. +- Refer to [Elastic OpenTelemetry Collector limitations](collector-limitations.md) for known limitations when using the Elastic OTel Collector. + +## Collect your logs and metrics + +Follow these steps to collect logs and metrics using the Elastic OTel collector: + +1. Open an [Elastic Cloud](cloud.elastic.co) deployment or a serverless Observability project. +1. To open the guided onboarding, either: + 1. In an Elastic Cloud deployment, open Kibana, and go to **Observability** → **Add Data**. + 1. In a serverless Observability project, go to **Add Data**. +1. Select **Collect and analyze logs**, and then select **OpenTelemetry**. +1. Select the appropriate platform, and complete the following: + 1. For **MacOS and Linux**, copy the command, open a terminal on your host, and run the command to download and configure the OpenTelemetry collector. + 1. For **Kubernetes**, download the manifest. +1. Copy the command under Step 2: + 1. For **MacOS and Linux**, run the command in your terminal to start the Elastic OTel collector. + 1. For **Kubernetes**, run the command from the directory where you downloaded the manifest to install the Elastic OTel Collector on every node of your cluster. + +Logs are collected from setup onward, so you won't see logs that occurred before starting the Elastic OTel Collector. +The default log path is `/var/log/*`. To update the path, modify `otel.yml`. + +Under **Visualize your data**, you'll see links to **Logs Explorer** to view your logs and **Hosts** to view your host metrics. \ No newline at end of file diff --git a/docs/images/elastic-otel-overview.png b/docs/images/elastic-otel-overview.png new file mode 100644 index 0000000..402c5ec Binary files /dev/null and b/docs/images/elastic-otel-overview.png differ diff --git a/docs/images/infrastructure-elastic-otel-flow.png b/docs/images/infrastructure-elastic-otel-flow.png new file mode 100644 index 0000000..6176bca Binary files /dev/null and b/docs/images/infrastructure-elastic-otel-flow.png differ diff --git a/docs/images/kubernetes-elastic-otel-flow.png b/docs/images/kubernetes-elastic-otel-flow.png new file mode 100644 index 0000000..0c8de33 Binary files /dev/null and b/docs/images/kubernetes-elastic-otel-flow.png differ diff --git a/docs/manual-configuration.md b/docs/manual-configuration.md new file mode 100644 index 0000000..5f10db5 --- /dev/null +++ b/docs/manual-configuration.md @@ -0,0 +1,37 @@ +# Manually configure the Elastic Distribution for OpenTelemetry Collector +Collecting logs and host metrics with the Elastic Distribution for OpenTelemetry Collector without using the [guided onboarding](guided-onboarding.md) requires some manual configuration. + +## Before you begin +The Elastic OTel collector has the following requirements and limitations: + +- The **Admin** role or higher is required to onboard system logs and metrics. To learn more, refer to [Assign user roles and privileges](https://www.elastic.co/docs/current/serverless/general/assign-user-roles). +- Root privileges on the host are required to run the OpenTelemetry collector used in this quickstart. +- The Elastic OTel Collector only works on Kubernetes, Linux, and MacOS systems. +- Refer to [Elastic OpenTelemetry Collector limitations](collector-limitations.md) for known limitations when using the Elastic Distribution for the OpenTelemetry collector. + +## Collect your logs and metrics + +To manually configure the Elastic OTel Collector, gather the following information: + +- **Your Elasticsearch endpoint**: From the help menu in Kibana or your serverless Observability project, select **Connection details** and copy the **Elasticsearch endpoint**. +- **API key**: + - **Kibana:** From the help menu, select **Connection details** and select **Create and manage API keys**. From the **API keys** page, select **Create API key**. Give your API key a name, select **Create API key**, and copy the new API key. + - **Serverless:** From the help menu, select **Connection details** and select the **API key** tab. Give your API key a name, select **Create API key**, and copy the new API key. + +Then manually configure the Elastic OTel Collector to collect logs and metrics on a MacOS or Linux system: + +1. Download and extract the standalone Elastic Agent for your platform. For more on downloading and extracting a standalone Elastic Agent, refer to the first step in [Install standalone Elastic Agents](https://www.elastic.co/guide/en/fleet/current/install-standalone-elastic-agent.html). +1. From the Elastic Agent base directory, go to the `otel_samples` directory. The `platformlogs_hostmetrics.yml` file has the configurations for the receivers, processors, and exporters needed to collect logs and host metrics. +1. Copy the content of the `platformlogs_hostmetrics.yml` file. +1. From the Elastic Agent base directory, open the `otel.yml` file, and replace the content with the copied content from `platformlogs_hostmetrics.yml`. +1. Find and update the following settings in the configuration: + - `file_storage.directory`: Set to the directory where you want to store you OpenTelemetry data. + - `elasticsearch.endpoint`: Set to your Elasticsearch endpoint you copied earlier. + - `elasticsearch.api_key`: Set to the API key you created earlier. +1. Run the OpenTelemetry collector with the following command: + ```console + ./elastic-agent otel --config otel.yml + ``` + +Logs are collected from setup onward, so you won't see logs that occurred before starting the collector. +The default log path is `/var/log/*`. Update the path in the `otel.yml` file. \ No newline at end of file