Skip to content

Commit

Permalink
review updates
Browse files Browse the repository at this point in the history
  • Loading branch information
mdbirnstiehl committed Aug 2, 2024
1 parent a615fa6 commit c5a1ba9
Show file tree
Hide file tree
Showing 6 changed files with 36 additions and 39 deletions.
23 changes: 12 additions & 11 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,26 +1,27 @@
# Elastic OpenTelemetry Distributions
# Elastic Distributions for OpenTelemetry

[OpenTelemetry](https://opentelemetry.io/docs/) is a vendor- and tool-neutral observability framework for collecting, processing, and exporting telemetry data.
The Elastic Distribution for OpenTelemetry (Elastic OTel Collector) is a supported distribution of the [OpenTelemetry Collector](https://github.com/open-telemetry/opentelemetry-collector) made up of mostly upstream components.
You can send your telemetry data to Elastic Observability using OpenTelemetry the following ways:

- Use the **Elastic Distribution for OpenTelemetry Collector**, an extension of the [OpenTelemetry Collector](https://github.com/open-telemetry/opentelemetry-collector), to collect and send logs and host metrics to [Elastic Cloud](https://cloud.elastic.co/).
- Use the **Elastic Distributions for OpenTelemetry**, extensions of select [OpenTelemetry language SDKs](https://opentelemetry.io/docs/languages/), to instrument your applications and send logs, traces, and metrics to [Elastic Cloud](https://cloud.elastic.co/). The following languages are currently supported: Java, .NET, Node.js, and Python.
- Configure your upstream [custom collector](https://opentelemetry.io/docs/collector/custom-collector/) or [contrib](https://github.com/open-telemetry/opentelemetry-collector-contrib) distribution for the collector to collect logs and metrics and send them to Elastic Observability.
- Collect and send logs and host metrics to [Elastic Cloud](https://cloud.elastic.co/) using the Elastic OTel Collector.
- Instrument your applications and send logs, traces, and metrics to [Elastic Cloud](https://cloud.elastic.co/) using the **Elastic Distributions for OpenTelemetry** extensions of select [OpenTelemetry language SDKs](https://opentelemetry.io/docs/languages/). The following languages are currently supported: Java, .NET, Node.js, and Python.
- Configure an upstream [custom collector](https://opentelemetry.io/docs/collector/custom-collector/) or [contrib](https://github.com/open-telemetry/opentelemetry-collector-contrib) distribution for the collector to collect logs and metrics and send them to Elastic Observability.

This diagram provides a quick overview on how the different components work together. Refer to the [components](docs/collector-components.md) for a more in-depth look.

![Diagram of the OpenTelemetry flow](docs/images/elastic-otel-overview.png)

## Get started with the Elastic Distribution for OpenTelemetry Collector
## Collect infrastructure data with the Elastic Distribution for OpenTelemetry Collector

These pages detail the components and how to configure the Elastic Distribution for OpenTelemetry Collector ("the collector").
These pages detail the components and how to configure the Elastic OTel Collector.

- [Components](docs/collector-components.md): Get details on the components used to receive, process, and export telemetry data.
- [Guided onboarding](docs/guided-onboarding.md): Use the guided onboarding in Kibana or in a serverless Observability project to send data using the collector.
- [Manual configurations](docs/manual-configuration.md): Manually configure the collector to send data to Elastic Observability.
- [Limitations](docs/collector-limitations.md): Understand the current limitations of the collector.
- [Guided onboarding](docs/guided-onboarding.md): Use the guided onboarding in Kibana or in a serverless Observability project to send data using the Elastic OTel Collector.
- [Manual configurations](docs/manual-configuration.md): Manually configure the Elastic OTel Collector to send data to Elastic Observability.
- [Limitations](docs/collector-limitations.md): Understand the current limitations of the Elastic OTel Collector.

## Collect application data with Elastic Distributions for OpenTelemetry
## Collect application data with Elastic Distributions for OpenTelemetry language SDKs

Elastic offers several distributions that extend [OpenTelemetry language SDKs](https://opentelemetry.io/docs/languages/). The following languages are currently supported:

Expand All @@ -31,4 +32,4 @@ Elastic offers several distributions that extend [OpenTelemetry language SDKs](h

## Configure an upstream collector for Elastic

[Configure a custom collector or a contrib distribution of the collector](docs/configure-upstream-collector.md): Configure a [custom collector](https://opentelemetry.io/docs/collector/custom-collector/) or [contrib](https://github.com/open-telemetry/opentelemetry-collector-contrib) distribution for the collector to collect logs and metrics and send them to Elastic Observability.
[Configure a custom collector or a contrib distribution of the OpenTelemetry collector](docs/configure-upstream-collector.md): Configure a [custom collector](https://opentelemetry.io/docs/collector/custom-collector/) or [contrib](https://github.com/open-telemetry/opentelemetry-collector-contrib) distribution for the OpenTelemetry collector to collect logs and metrics and send them to Elastic Observability.
9 changes: 4 additions & 5 deletions docs/collector-components.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# Elastic OpenTelemetry Collector components
# Elastic Distribution for OpenTelemetry Collector components

The OpenTelemetry Collector uses the following components to receive, process, and export telemetry data:

Expand All @@ -7,7 +7,7 @@ The OpenTelemetry Collector uses the following components to receive, process, a
- [Exporters](collector-components.md#exporters): Send data to the backends or destinations.
- [Extensions](collector-components.md#extensions): Provide additional functionalities and capabilities.

The default configurations of the Elastic Distribution for the OpenTelemetry Collector follows these flows.
The default configurations of the Elastic OTel Collector follows these flows.

**MacOS and Linux Host metrics:**

Expand Down Expand Up @@ -65,7 +65,7 @@ flowchart LR
style eight fill:#e6f9f7,stroke:#333,stroke-width:1px
```

**Kubernetes logs**
**Kubernetes, MacOS, and Linux logs**
```mermaid
flowchart LR
one["`File log receiver`"]
Expand All @@ -84,7 +84,7 @@ flowchart LR
style six fill: #e6f9f7,stroke:#333,stroke-width:1px
```

Refer to the following tables for more information on the components supported by the Elastic Distribution for the OpenTelemetry Collector.
Refer to the following tables for more information on the components included in the Elastic Distribution for the OpenTelemetry Collector.
Follow the links for OpenTelemetry documentation with more configuration details for each component.
To set up the collector, go to the [guided onboarding](guided-onboarding.md) or [manual configuration](manual-configuration.md) to get started.

Expand All @@ -104,7 +104,6 @@ To set up the collector, go to the [guided onboarding](guided-onboarding.md) or

| Component | Description |
|---|---|
| [`elasticinframetricsprocessor`](https://github.com/elastic/opentelemetry-collector-components/blob/processor/elasticinframetricsprocessor/v0.7.1/processor/elasticinframetricsprocessor/README.md) | Processes infrastructure metrics to enhance and convert them for Elasticsearch. |
| [`attributesprocessor`](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/processor/attributesprocessor/v0.105.0/processor/attributesprocessor/README.md) | Modifies telemetry data attributes. |
| [`filterprocessor`](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/processor/filterprocessor/v0.105.0/processor/filterprocessor/README.md) | Filters telemetry data to include or exclude specific data points. |
| [`k8sattributesprocessor`](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/processor/k8sattributesprocessor/v0.105.0/processor/k8sattributesprocessor/README.md) | Enhances telemetry data with Kubernetes-specific metadata. |
Expand Down
6 changes: 3 additions & 3 deletions docs/collector-limitations.md
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
# Elastic OpenTelemetry Collector limitations
# Elastic Distribution for OpenTelemetry Collector limitations

The Elastic Distribution for the OpenTelemetry Collector has the following limitations:
The Elastic OTel Collector has the following limitations:

- `host.network.*` metrics aren't present from the OpenTelemetry side.
- Because of an upstream limitation, `host.network.*` metrics aren't present from the OpenTelemetry side.
- `process.state` isn't present in the OpenTelemetry host metric. It's set to a dummy value of **Unknown** in the **State** column of the host processes table.
- The Elasticsearch exporter handles the metadata fields, but **Host OS version** and **Operating system** may show as "N/A" and **Host IP** may show different values.
- The CPU scraper needs to be enabled to collect the `systm.load.cores` metric, which affects the **Normalized Load** column in the **Hosts** table and the **Normalized Load** visualization on the host detailed view.
9 changes: 3 additions & 6 deletions docs/configure-upstream-collector.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,12 +2,9 @@

You can configure an upstream collector, like a [custom collector](https://opentelemetry.io/docs/collector/custom-collector/) or [contrib](https://github.com/open-telemetry/opentelemetry-collector-contrib) distribution for the collector, to collect logs and metrics and send them to Elastic Observability.

<!-- We might want to add some pros and cons of using an upstream collector -->

For a more seamless experience, use the Elastic distribution for the OpenTelemetry collector.
Refer to the [guided onboarding](guided-onboarding.md) docs or the [manual configuration](manual-configuration.md) docs for more on configuring the Elastic distribution.

## Upstream collector configuration example
Use the following configuration example as a reference when configuring an upstream collector.

<!-- Add config example-->
## Upstream collector configuration examples
Refer to the OpenTelemetry documentation on [building a custom collector](https://opentelemetry.io/docs/collector/custom-collector/) for more on creating an upstream collector.
Use the Elastic [example configurations](https://github.com/elastic/elastic-agent/tree/main/internal/pkg/otel/samples) as a reference when configuring your upstream collector.
16 changes: 8 additions & 8 deletions docs/guided-onboarding.md
Original file line number Diff line number Diff line change
@@ -1,18 +1,18 @@
# Collect logs and metrics using the guided onboarding
The guided onboarding in Kibana or in a serverless Observability project walks you through collecting logs and metrics using the Elastic OpenTelemetry Collector.
If you prefer manually configuring the Elastic OpenTelemetry collector, refer to [Manually configure the collector](manual-configuration.md).
The guided onboarding in Kibana or in a serverless Observability project walks you through collecting logs and metrics using the Elastic OTel Collector.
For manual configuration, refer to [Manually configure the Elastic OTel Collector](manual-configuration.md).

## Before you begin
The onboarding has the following requirements and limitations:

- The **Admin** role or higher is required to onboard system logs and metrics. To learn more, refer to [Assign user roles and privileges](https://www.elastic.co/docs/current/serverless/general/assign-user-roles).
- Root privileges on the host are required to run the OpenTelemetry collector used in this quickstart.
- The collector only works on Kubernetes, Linux, and MacOS systems.
- Refer to [Elastic OpenTelemetry Collector limitations](collector-limitations.md) for known limitations when using Elastic Distribution for the OpenTelemetry collector.
- The Elastic OTel Collector only works on Kubernetes, Linux, and MacOS systems.
- Refer to [Elastic OpenTelemetry Collector limitations](collector-limitations.md) for known limitations when using the Elastic OTel Collector.

## Collect your logs and metrics

Follow these steps to collect logs and metrics using the Elastic OpenTelemetry collector
Follow these steps to collect logs and metrics using the Elastic OTel collector

1. Open an [Elastic Cloud](cloud.elastic.co) deployment or a serverless Observability project.
1. To open the guided onboarding, either:
Expand All @@ -23,10 +23,10 @@ Follow these steps to collect logs and metrics using the Elastic OpenTelemetry c
1. For **MacOS and Linux**, copy the command, open a terminal on your host, and run the command to download and configure the OpenTelemetry collector.
1. For **Kubernetes**, download the manifest.
1. Copy the command under Step 2:
1. For **MacOS and Linux**, run the command in your terminal to start the OpenTelemetry collector.
1. For **Kubernetes**, run the command from the directory where you downloaded the manifest to install the collector on every node of your cluster.
1. For **MacOS and Linux**, run the command in your terminal to start the Elastic OTel collector.
1. For **Kubernetes**, run the command from the directory where you downloaded the manifest to install the Elastic OTel Collector on every node of your cluster.

Logs are collected from setup onward, so you won't see logs that occurred before starting the collector.
Logs are collected from setup onward, so you won't see logs that occurred before starting the Elastic OTel Collector.
The default log path is `/var/log/*`. To update the path, modify `otel.yml`.

Under **Visualize your data**, you'll see links to **Logs Explorer** to view your logs and **Hosts** to view your host metrics.
12 changes: 6 additions & 6 deletions docs/manual-configuration.md
Original file line number Diff line number Diff line change
@@ -1,24 +1,24 @@
# Manually configure the collector
Collecting logs and host metrics with the Elastic Distribution for the OpenTelemetry Collector without using the [guided onboarding](guided-onboarding.md) requires some manual configuration.
# Manually configure the Elastic Distribution for OpenTelemetry Collector
Collecting logs and host metrics with the Elastic OTel Collector without using the [guided onboarding](guided-onboarding.md) requires some manual configuration.

## Before you begin
The onboarding has the following requirements and limitations:
The Elastic OTel collector has the following requirements and limitations:

- The **Admin** role or higher is required to onboard system logs and metrics. To learn more, refer to [Assign user roles and privileges](https://www.elastic.co/docs/current/serverless/general/assign-user-roles).
- Root privileges on the host are required to run the OpenTelemetry collector used in this quickstart.
- The collector only works on Kubernetes, Linux, and MacOS systems.
- The Elastic OTel Collector only works on Kubernetes, Linux, and MacOS systems.
- Refer to [Elastic OpenTelemetry Collector limitations](collector-limitations.md) for known limitations when using the Elastic Distribution for the OpenTelemetry collector.

## Collect your logs and metrics

To manually configure the collector, gather the following information:
To manually configure the Elastic OTel Collector, gather the following information:

- **Your Elasticsearch endpoint**: From the help menu in Kibana or your serverless Observability project, select **Connection details** and copy the **Elasticsearch endpoint**.
- **API key**:
- **Kibana:** From the help menu, select **Connection details** and select **Create and manage API keys**. From the **API keys** page, select **Create API key**. Give your API key a name, select **Create API key**, and copy the new API key.
- **Serverless:** From the help menu, select **Connection details** and select the **API key** tab. Give your API key a name, select **Create API key**, and copy the new API key.

Then manually configure the Elastic OpenTelemetry Collector to collect logs and metrics on a MacOS or Linux system:
Then manually configure the Elastic OTel Collector to collect logs and metrics on a MacOS or Linux system:

1. Download and extract the standalone Elastic Agent for your platform. For more on downloading and extracting a standalone Elastic Agent, refer to the first step in [Install standalone Elastic Agents](https://www.elastic.co/guide/en/fleet/current/install-standalone-elastic-agent.html).
1. From the Elastic Agent base directory, go to the `otel_samples` directory. The `platformlogs_hostmetrics.yml` file has the configurations for the receivers, processors, and exporters needed to collect logs and host metrics.
Expand Down

0 comments on commit c5a1ba9

Please sign in to comment.