Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docs(thanos): add migration doc for thanos storage clients #16133

Merged
merged 11 commits into from
Feb 11, 2025
3 changes: 2 additions & 1 deletion docs/sources/configure/examples/_index.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,4 +10,5 @@ weight: 800
The following pages contain examples of how to configure Grafana Loki.

- [Configuration snippets and ready-to-use configuration examples]({{< relref "./configuration-examples" >}}).
- [Deploy a query frontend on a existing cluster]({{< relref "./query-frontend" >}}).
- [Deploy a query frontend on a existing cluster]({{< relref "./query-frontend" >}}).
- [Configuration examples for using Thanos-based storage clients](./thanos-storage-configs).
79 changes: 79 additions & 0 deletions docs/sources/configure/examples/thanos-storage-configs.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,79 @@
---
title: "Configuration examples for using Thanos-based storage clients"
menuTitle: Thanos storage examples
description: "Minimal examples for using Thanos-based S3, GCS, Azure, and filesystem clients in Grafana Loki."
weight: 100
---

# Configuration examples for using Thanos-based storage clients

Use these examples as a starting point for configuring [Thanos based object storage clients](https://grafana.com/docs/loki/<LOKI_VERSION>/configure/#thanos_object_store_config) in Grafana Loki.

## GCS example
```yaml
storage_config:
use_thanos_objstore: true
object_store:
gcs:
bucket_name: my-gcs-bucket

# JSON either from a Google Developers Console client_credentials.json file,
JStickler marked this conversation as resolved.
Show resolved Hide resolved
# or a Google Developers service account key. Needs to be valid JSON, not a
# filesystem path. If empty, fallback to Google default logic:
# 1. A JSON file whose path is specified by the GOOGLE_APPLICATION_CREDENTIALS
# environment variable. For workload identity federation, refer to
# https://cloud.google.com/iam/docs/how-to#using-workload-identity-federation
# on how to generate the JSON configuration file for on-prem/non-Google cloud
# platforms.
# 2. A JSON file in a location known to the gcloud command-line tool:
# $HOME/.config/gcloud/application_default_credentials.json.
# 3. On Google Compute Engine it fetches credentials from the metadata server.
service_account: |-
{
"type": "service_account",
"project_id": "project",
"private_key_id": "abcdefghijklmnopqrstuvwxyz12345678906666",
"private_key": "-----BEGIN PRIVATE KEY-----\...\n-----END PRIVATE KEY-----\n",
"client_email": "[email protected]",
"client_id": "123456789012345678901",
"auth_uri": "https://accounts.google.com/o/oauth2/auth",
"token_uri": "https://oauth2.googleapis.com/token",
"auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
"client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/[email protected]"
}
```

## S3 example
```yaml
storage_config:
use_thanos_objstore: true
object_store:
s3:
bucket_name: my-s3-bucket
endpoint: s3.us-west-2.amazonaws.com
region: us-west-2
# You can either declare the access key and secret in the config or
# use environment variables AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY which will be picked up by the AWS SDK.
access_key_id: access-key-id
secret_access_key: secret-access-key
```

## Azure example
```yaml
storage_config:
use_thanos_objstore: true
object_store:
azure:
account_name: myaccount
account_key: ${SECRET_ACCESS_KEY} # loki expands environment variables
container_name: example-container
```

## Filesystem example
```yaml
storage_config:
use_thanos_objstore: true
object_store:
filesystem:
dir: /var/loki/chunks
```
1 change: 1 addition & 0 deletions docs/sources/setup/migrate/_index.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,3 +12,4 @@ This section contains instructions for migrating from one Loki implementation to
- [Migrate]({{< relref "./migrate-to-tsdb" >}}) to TSDB index.
- [Migrate]({{< relref "./migrate-from-distributed" >}}) from the `Loki-distributed` Helm chart to the `loki` Helm chart.
- [Migrate]({{< relref "./migrate-to-three-scalable-targets" >}}) from the two target Helm chart to the three target scalable configuration Helm chart.
- [Migrate]({{< relref "./migrate-storage-clients" >}}) from the legacy storage clients to the Thanos object storage client.
233 changes: 233 additions & 0 deletions docs/sources/setup/migrate/migrate-storage-clients/_index.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,233 @@
---
title: Migrate to Thanos storage clients
menuTitle: Migrate to Thanos storage clients
description: Migration guide for moving from existing storage clients to Thanos storage clients.
weight:
---
# Migrate to Thanos storage clients

Loki release 3.4 introduces new object storage clients based on the [Thanos Object Storage Client Go module](https://github.com/thanos-io/objstore).

One of the reasons for making this change is to have a consistent storage configuration across Grafana Loki, Mimir and other telemetry databases from Grafana Labs. If you are already using Grafana Mimir or Pyroscope, you can reuse the storage configuration for setting up Loki.

This is an opt-in feature with the Loki 3.4 release. In a future release, Thanos will become the default way of configuring storage and the existing storage clients will be deprecated.

{{< admonition type="note" >}}
The new storage configuration deviates from the existing format. The following sections describe the changes in detail for each provider.
Refer to the [Thanos storage configuration reference](https://grafana.com/docs/loki/<LOKI_VERSION>/configure/#thanos_object_store_config) to view the complete list of supported storage providers and their configuration options.
{{< /admonition >}}

### Enable the new storage clients

1. Enable Thanos storage clients by setting `use_thanos_objstore` to `true` in the `storage_config` section or by setting the `-use-thanos-objstore` flag to true. When enabled, configuration under `storage_config.object_store` takes effect instead of existing storage configurations.

```yaml
# Uses the new storage clients for connecting to gcs backend
storage_config:
use_thanos_objstore: true # enable the new storage clients
object_store:
gcs:
bucket_name: "example-bucket"
```
ashwanthgoli marked this conversation as resolved.
Show resolved Hide resolved

1. As an alternative, you can also configure the new clients in the common `storage` section if you prefer to use the `common` config section.

```yaml
storage_config:
use_thanos_objstore: true # enable the new storage clients
common:
storage:
object_store:
gcs:
bucket_name: "example-bucket"
```

1. Ruler storage should be configured under the `ruler_storage` section when using the new storage clients.

```yaml
storage_config:
use_thanos_objstore: true # enable the new storage clients
ruler_storage:
backend: gcs
gcs:
bucket_name: "example-bucket"
```

1. If you are using `store.object-prefix` flag or the corresponding `object_prefix` YAML setting, you'll need to update your configuration to use the new `object_store.storage-prefix` flag or the corresponding `storage_prefix` YAML setting.

```yaml
# Example configuration to prefix all objects with "prefix"
storage_config:
use_thanos_objstore: true # enable the new storage clients
object_store:
storage_prefix: "prefix"
```

### GCS Storage Migration

When migrating from the existing [Google Cloud Storage (GCS)](https://grafana.com/docs/loki/<LOKI_VERSION>/configure/#gcs_storage_config) storage client to the new Thanos-based client, you'll need to update your configuration parameters as follows:

{{< responsive-table >}}
| Existing Parameter | New Parameter | Required Changes |
|---------------------|-----------------------|--------------------------------------------------------------------------------------------------------------|
| `bucket_name` | `bucket_name` | No changes required |
| `service_account` | `service_account` | No changes required |
| `chunk_buffer_size` | `chunk_buffer_size` | No changes required |
| `enable_retries` | `max_retries` | Replace `enable_retries` (bool) with `max_retries` (int). Set a value > 1 to enable retries, or 1 to disable |
| `request_timeout` | Removed | Remove parameter |
| `enable_opencensus` | Removed | Remove parameter |
| `enable_http2` | Removed | Remove parameter |
{{< /responsive-table >}}

**Example configuration migration (GCS):**

_**Existing configuration:**_

```yaml
storage_config:
gcs:
bucket_name: example-bucket
chunk_buffer_size: 10MB
enable_retries: true
```

_**New configuration**_ (Thanos-based):

```yaml
storage_config:
use_thanos_objstore: true
object_store:
gcs:
bucket_name: example-bucket
chunk_buffer_size: 10MB
max_retries: 5
```

### Amazon S3 Storage Migration

When migrating from the existing [Amazon S3](https://grafana.com/docs/loki/<LOKI_VERSION>/configure/#aws_storage_config) storage client to the new Thanos-based client, update or remove parameters as follows:

{{< responsive-table >}}
| Existing Parameter | New Parameter | Required Changes |
|---------------------|-----------------------|------------------------------------------------------------------------------------------------------------------------------------|
| `bucket_names` | `bucket_name` | Rename this parameter. If you previously used multiple buckets, you must consolidate to a single bucket (Thanos supports only one). |
| `endpoint` | `endpoint` | No changes required. |
| `region` | `region` | No changes required. |
| `access_key_id` | `access_key_id` | No changes required. |
| `secret_access_key` | `secret_access_key` | No changes required. |
| `session_token` | `session_token` | No changes required. |
| `insecure` | `insecure` | No changes required. |
| `disable_dualstack` | `dualstack_enabled` | Renamed and inverted. If you had `disable_dualstack: false`, set `dualstack_enabled: true`. |
| `storage_class` | `storage_class` | No changes required. |
| `s3` | Removed | Remove or replace with `endpoint` if you used the URL-based setup. |
| `S3ForcePathStyle` | Removed or replaced | If you need path-based addressing, set `bucket_lookup_type: path` in the new config. Otherwise, remove it. |
| `signature_version` | Removed | Remove parameter. Thanos always uses Signature Version 4 (V4). |
| `http_config` | `http` | Move subfields (such as timeouts, CA file, etc.) into the `http:` block in the Thanos configuration. |
| `sse` | `sse` | Migrate any SSE settings (e.g., `type`, `kms_key_id`) into the `sse:` block in the Thanos configuration. |
| `backoff_config` | `max_retries`| Replace the advanced backoff settings with a single integer (`max_retries`). Set to 1 to disable retries, or a higher value to enable them. |
{{< /responsive-table >}}

**Example configuration migration (S3):**

_**Existing configuration**_

```yaml
storage_config:
aws:
bucket_names: my-bucket1,my-bucket2 # multiple buckets no longer supported
endpoint: s3.amazonaws.com
region: us-west-2
access_key_id: example-key
secret_access_key: example-secret
signature_version: v4
disable_dualstack: true
storage_class: STANDARD
http_config:
timeout: 1m
insecure_skip_verify: false
# ...
backoff_config:
max_retries: 5
sse:
type: SSE-KMS
kms_key_id: mySSEKey
```

_**New configuration** (Thanos-based)_

```yaml
storage_config:
use_thanos_objstore: true
object_store:
s3:
bucket_name: my-bucket1 # single bucket
endpoint: s3.amazonaws.com
region: us-west-2
access_key_id: example-key
secret_access_key: example-secret
dualstack_enabled: false # was disable_dualstack: true
storage_class: STANDARD
max_retries: 5
http:
insecure_skip_verify: false
sse:
type: SSE-KMS
kms_key_id: mySSEKey
```

For more advanced configuration options (such as `list_objects_version`, `bucket_lookup_type`, etc.), see the [Thanos S3 configuration reference](https://grafana.com/docs/loki/<LOKI_VERSION>/configure/#thanos_object_store_config).

### Azure Storage Migration

When migrating from the existing [Azure](https://grafana.com/docs/loki/<LOKI_VERSION>/configure/#azure_storage_config) storage client to the new Thanos-based client, no changes are required if you are using the following parameters:

{{< responsive-table >}}
| Existing Parameter | New Parameter | Required Changes |
|-----------------|---------------|------------------|
| `account_name` | `account_name` | No changes required |
| `account_key` | `account_key` | No changes required |
| `container_name` | `container_name` | No changes required |
| `endpoint_suffix` | `endpoint_suffix` | No changes required |
| `user_assigned_id` | `user_assigned_id` | No changes required |
| `connection_string` | `connection_string` | No changes required |
| `max_retries` | `max_retries` | No changes required |
| `chunk_delimiter` | `chunk_delimiter` | No changes required |
{{< /responsive-table >}}

If you are using an authentication method other than storage account key or user-assigned managed identity, you'll have to pass the neccessary credetials using environment variables.
For more details, refer to [Azure Identity Client Module for Go](https://pkg.go.dev/github.com/Azure/azure-sdk-for-go/sdk/azidentity).

### Filesystem Storage Migration

When migrating from the existing [Filesystem storage](https://grafana.com/docs/loki/<LOKI_VERSION>/configure/#local_storage_config)
client to the new Thanos-based client, update or remove parameters as follows:

{{< responsive-table >}}
| Existing Parameter | New Parameter | Required Changes |
|-----------------|---------------|-------------------------------|
| `directory` | `dir` | Rename `directory` to `dir`. |
{{< /responsive-table >}}

**Example configuration migration (Filesystem):**

_**Existing configuration** (`FSConfig`)_

```yaml
storage_config:
filesystem:
directory: /var/loki/chunks
```

_**New configuration** (Thanos-based)_

```yaml
storage_config:
use_thanos_objstore: true
object_store:
filesystem:
dir: /var/loki/chunks
```

{{< admonition type="note" >}}
For providers not listed here, refer to the [Thanos storage configuration reference](https://grafana.com/docs/loki/<LOKI_VERSION>/configure/#thanos_object_store_config).
{{< /admonition >}}