From 6d970e5a790fe2066a686ca27cc99cf4a49212f0 Mon Sep 17 00:00:00 2001 From: Ashwanth Goli Date: Thu, 6 Feb 2025 11:13:38 +0530 Subject: [PATCH 1/9] docs(thanos): add migration doc for thanos storage clients --- docs/sources/setup/migrate/_index.md | 1 + .../migrate/migrate-storage-clients/_index.md | 155 ++++++++++++++++++ 2 files changed, 156 insertions(+) create mode 100644 docs/sources/setup/migrate/migrate-storage-clients/_index.md diff --git a/docs/sources/setup/migrate/_index.md b/docs/sources/setup/migrate/_index.md index 034791b38040d..f2982d66ddfa8 100644 --- a/docs/sources/setup/migrate/_index.md +++ b/docs/sources/setup/migrate/_index.md @@ -12,3 +12,4 @@ This section contains instructions for migrating from one Loki implementation to - [Migrate]({{< relref "./migrate-to-tsdb" >}}) to TSDB index. - [Migrate]({{< relref "./migrate-from-distributed" >}}) from the `Loki-distributed` Helm chart to the `loki` Helm chart. - [Migrate]({{< relref "./migrate-to-three-scalable-targets" >}}) from the two target Helm chart to the three target scalable configuration Helm chart. +- [Migrate]({{< relref "./migrate-storage-clients" >}}) from the legacy storage clients to the Thanos object storage client. \ No newline at end of file diff --git a/docs/sources/setup/migrate/migrate-storage-clients/_index.md b/docs/sources/setup/migrate/migrate-storage-clients/_index.md new file mode 100644 index 0000000000000..47f2bb6d4ed96 --- /dev/null +++ b/docs/sources/setup/migrate/migrate-storage-clients/_index.md @@ -0,0 +1,155 @@ +--- +title: Migrate storage clients +menuTitle: Migrate storage clients +description: Links to Grafana Loki migrate storage clients documentation. +weight: +--- +# Migrate storage clients + +Loki 3.4 release introduces new object storage clients based on [Thanos Object Storage Client Go module](https://github.com/thanos-io/objstore). + +One of the reasons for making this change is to have a consistent storage configuration across LGTM+ stack. If you are already using Grafana Mimir or Pyroscope, you can reuse the storage configuration for setting up Loki. + +This is an opt-in feature with Loki 3.4 release. This would be made the default way of configuring storage in future releases and the existing storage clients will be deprecated. + +The new storage configuration deviates from the existing format. The following sections describe the changes in detail for each provider. +Refer to the [thanos storage reference](https://grafana.com/docs/loki//configure/#thanos_object_store_config) to view the complete list of supported storage providers and their configuration options. + +### Enabling the new storage clients +1. Enable thanos storage clients by setting `use_thanos_objstore` to `true` in the `storage_config` section or by setting `-use-thanos-objstore` flag to true. +1. When enabled, configuration under `storage_config.object_store` takes effect instead of legacy storage configs + ```yaml + # Uses the new storage clients for connecting to gcs backend + storage_config: + use_thanos_objstore: true # enable the new storage clients + object_store: + gcs: + bucket: "example-bucket" + ``` +1. You can also configure the new clients in the common storage section if you prefer to use the common config section. + ```yaml + storage_config: + use_thanos_objstore: true # enable the new storage clients + common: + storage: + object_store: + gcs: + bucket: "example-bucket" + ``` +1. Ruler storage should be configured under `ruler_storage` section when using the new storage clients. + ```yaml + storage_config: + use_thanos_objstore: true # enable the new storage clients + ruler_storage: + backend: gcs + gcs: + bucket: "example-bucket" + ``` + +{{< admonition type="note" >}} +Following sections document the steps for migrating the clients. Only some of the commonly used parameters are listed here. +For a complete list of supported parameters, refer to the [thanos storage reference](https://grafana.com/docs/loki//configure/#thanos_object_store_config) and compare that to what you are using in your setup. +{{< /admonition >}} + +### GCS Storage Migration + +When migrating from the legacy GCS storage client to the new Thanos-based client, you'll need to update your configuration parameters as follows: + +{{< responsive-table >}} +| Legacy Parameter | New Parameter | Required Changes | +|-----------------|---------------|------------------| +| `bucket_name` | `bucket_name` | No changes required | +| `chunk_buffer_size` | `chunk_size_bytes` | Rename parameter to `chunk_size_bytes` | +| `enable_retries` | `enable_retries` | No changes required | +{{< /responsive-table >}} + +**Example configuration migration:** + +Legacy configuration: +```yaml +storage_config: + gcs: + bucket_name: example-bucket + chunk_buffer_size: 10MB + enable_retries: true +``` + +New configuration: +```yaml +storage_config: + use_thanos_objstore: true +object_store: + gcs: + bucket_name: example-bucket + chunk_size_bytes: 10MB + enable_retries: true +``` + +### S3 Storage Migration + +When migrating from the legacy S3 storage client to the new Thanos-based client, you'll need to update your configuration parameters as follows: + +{{< responsive-table >}} +| Legacy Parameter | New Parameter | Required Changes | +|-----------------|---------------|------------------| +| `endpoint` | `endpoint` | No changes required | +| `region` | `region` | No changes required | +| `bucket_names` | `bucket_name` | Rename parameter to `bucket_name` | +| `access_key_id` | `access_key_id` | No changes required | +| `secret_access_key` | `secret_access_key` | No changes required | +| `session_token` | `session_token` | No changes required | +| `insecure` | `insecure` | No changes required | +| `disable_dualstack` | `dualstack_enabled` | Rename parameter to `dualstack_enabled` and invert logic | +| `storage_class` | `storage_class` | No changes required | +{{< /responsive-table >}} + +**Example configuration migration:** + +Legacy configuration: +```yaml +storage_config: + aws: + endpoint: s3.amazonaws.com + region: us-west-2 + bucket_names: example-bucket + access_key_id: example-key + secret_access_key: example-secret + insecure: false + disable_dualstack: true + storage_class: STANDARD +``` + +New configuration: +```yaml +storage_config: + use_thanos_objstore: true +object_store: + s3: + endpoint: s3.amazonaws.com + region: us-west-2 + bucket_name: example-bucket + access_key_id: example-key + secret_access_key: example-secret + insecure: false + dualstack_enabled: false + storage_class: STANDARD +``` + +### Azure Storage Migration + +When migrating from the legacy Azure storage client to the new Thanos-based client, no changes are required if you are using the following parameters: + +{{< responsive-table >}} +| Legacy Parameter | New Parameter | Required Changes | +|-----------------|---------------|------------------| +| `account_name` | `account_name` | No changes required | +| `account_key` | `account_key` | No changes required | +| `container_name` | `container_name` | No changes required | +| `endpoint_suffix` | `endpoint_suffix` | No changes required | +| `user_assigned_id` | `user_assigned_id` | No changes required | +| `connection_string` | `connection_string` | No changes required | +| `max_retries` | `max_retries` | No changes required | +{{< /responsive-table >}} + +If you are using an authentication method other than storage account key or user-assigned managed identity, you'll have to pass the neccessary credetials using environment variables. +For more details, refer to [Azure Identity Client Module for Go](https://pkg.go.dev/github.com/Azure/azure-sdk-for-go/sdk/azidentity). From 9dc4f3679290e034b8860a00f2e50b11fce27f28 Mon Sep 17 00:00:00 2001 From: Ashwanth Goli Date: Thu, 6 Feb 2025 19:17:27 +0530 Subject: [PATCH 2/9] fixup! docs(thanos): add migration doc for thanos storage clients --- .../setup/migrate/migrate-storage-clients/_index.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/sources/setup/migrate/migrate-storage-clients/_index.md b/docs/sources/setup/migrate/migrate-storage-clients/_index.md index 47f2bb6d4ed96..2d4c1662cf806 100644 --- a/docs/sources/setup/migrate/migrate-storage-clients/_index.md +++ b/docs/sources/setup/migrate/migrate-storage-clients/_index.md @@ -1,10 +1,10 @@ --- -title: Migrate storage clients -menuTitle: Migrate storage clients -description: Links to Grafana Loki migrate storage clients documentation. +title: Migrate to Thanos storage clients +menuTitle: Migrate to Thanos storage clients +description: Migration guide for moving from legacy storage clients to Thanos storage clients. weight: --- -# Migrate storage clients +# Migrate to Thanos storage clients Loki 3.4 release introduces new object storage clients based on [Thanos Object Storage Client Go module](https://github.com/thanos-io/objstore). From 8298b38ade31d23634d72da8a03e36a8c2106e99 Mon Sep 17 00:00:00 2001 From: Ashwanth Date: Fri, 7 Feb 2025 10:40:35 +0530 Subject: [PATCH 3/9] Apply suggestions from code review Co-authored-by: J Stickler Signed-off-by: Ashwanth --- .../migrate/migrate-storage-clients/_index.md | 27 ++++++++++--------- 1 file changed, 14 insertions(+), 13 deletions(-) diff --git a/docs/sources/setup/migrate/migrate-storage-clients/_index.md b/docs/sources/setup/migrate/migrate-storage-clients/_index.md index 2d4c1662cf806..cf20f8090170e 100644 --- a/docs/sources/setup/migrate/migrate-storage-clients/_index.md +++ b/docs/sources/setup/migrate/migrate-storage-clients/_index.md @@ -6,18 +6,19 @@ weight: --- # Migrate to Thanos storage clients -Loki 3.4 release introduces new object storage clients based on [Thanos Object Storage Client Go module](https://github.com/thanos-io/objstore). +Loki release 3.4 introduces new object storage clients based on the [Thanos Object Storage Client Go module](https://github.com/thanos-io/objstore). One of the reasons for making this change is to have a consistent storage configuration across LGTM+ stack. If you are already using Grafana Mimir or Pyroscope, you can reuse the storage configuration for setting up Loki. -This is an opt-in feature with Loki 3.4 release. This would be made the default way of configuring storage in future releases and the existing storage clients will be deprecated. +This is an opt-in feature with the Loki 3.4 release. In a future release, Thanos will become the default way of configuring storage and the existing storage clients will be deprecated. The new storage configuration deviates from the existing format. The following sections describe the changes in detail for each provider. -Refer to the [thanos storage reference](https://grafana.com/docs/loki//configure/#thanos_object_store_config) to view the complete list of supported storage providers and their configuration options. +Refer to the [Thanos storage configuration reference](https://grafana.com/docs/loki//configure/#thanos_object_store_config) to view the complete list of supported storage providers and their configuration options. + +### Enable the new storage clients + +1. Enable Thanos storage clients by setting `use_thanos_objstore` to `true` in the `storage_config` section or by setting the `-use-thanos-objstore` flag to true. When enabled, configuration under `storage_config.object_store` takes effect instead of legacy storage configurations. -### Enabling the new storage clients -1. Enable thanos storage clients by setting `use_thanos_objstore` to `true` in the `storage_config` section or by setting `-use-thanos-objstore` flag to true. -1. When enabled, configuration under `storage_config.object_store` takes effect instead of legacy storage configs ```yaml # Uses the new storage clients for connecting to gcs backend storage_config: @@ -26,7 +27,7 @@ Refer to the [thanos storage reference](https://grafana.com/docs/loki/}} Following sections document the steps for migrating the clients. Only some of the commonly used parameters are listed here. -For a complete list of supported parameters, refer to the [thanos storage reference](https://grafana.com/docs/loki//configure/#thanos_object_store_config) and compare that to what you are using in your setup. +For a complete list of supported parameters, refer to the [Thanos storage configuration reference](https://grafana.com/docs/loki//configure/#thanos_object_store_config) and compare that to what you are using in your setup. {{< /admonition >}} ### GCS Storage Migration -When migrating from the legacy GCS storage client to the new Thanos-based client, you'll need to update your configuration parameters as follows: +When migrating from the legacy [Google Cloud Storage (GCS)](https://grafana.com/docs/loki//configure/#gcs_storage_config) storage client to the new Thanos-based client, you'll need to update your configuration parameters as follows: {{< responsive-table >}} | Legacy Parameter | New Parameter | Required Changes | @@ -85,9 +86,9 @@ object_store: enable_retries: true ``` -### S3 Storage Migration +### Amazon S3 Storage Migration -When migrating from the legacy S3 storage client to the new Thanos-based client, you'll need to update your configuration parameters as follows: +When migrating from the legacy [Amazon S3](https://grafana.com/docs/loki//configure/#aws_storage_config) storage client to the new Thanos-based client, you'll need to update your configuration parameters as follows: {{< responsive-table >}} | Legacy Parameter | New Parameter | Required Changes | @@ -137,7 +138,7 @@ object_store: ### Azure Storage Migration -When migrating from the legacy Azure storage client to the new Thanos-based client, no changes are required if you are using the following parameters: +When migrating from the legacy [Azure](https://grafana.com/docs/loki//configure/#azure_storage_config) storage client to the new Thanos-based client, no changes are required if you are using the following parameters: {{< responsive-table >}} | Legacy Parameter | New Parameter | Required Changes | From 3f705bf2b95b9da8a680173e76120c24984e1f23 Mon Sep 17 00:00:00 2001 From: Ashwanth Goli Date: Fri, 7 Feb 2025 11:36:34 +0530 Subject: [PATCH 4/9] make corrections to the example configs + update table entries --- .../migrate/migrate-storage-clients/_index.md | 179 ++++++++++++------ 1 file changed, 124 insertions(+), 55 deletions(-) diff --git a/docs/sources/setup/migrate/migrate-storage-clients/_index.md b/docs/sources/setup/migrate/migrate-storage-clients/_index.md index cf20f8090170e..b2b1a1f84c956 100644 --- a/docs/sources/setup/migrate/migrate-storage-clients/_index.md +++ b/docs/sources/setup/migrate/migrate-storage-clients/_index.md @@ -1,14 +1,14 @@ --- title: Migrate to Thanos storage clients menuTitle: Migrate to Thanos storage clients -description: Migration guide for moving from legacy storage clients to Thanos storage clients. +description: Migration guide for moving from existing storage clients to Thanos storage clients. weight: --- # Migrate to Thanos storage clients Loki release 3.4 introduces new object storage clients based on the [Thanos Object Storage Client Go module](https://github.com/thanos-io/objstore). -One of the reasons for making this change is to have a consistent storage configuration across LGTM+ stack. If you are already using Grafana Mimir or Pyroscope, you can reuse the storage configuration for setting up Loki. +One of the reasons for making this change is to have a consistent storage configuration across Grafana Loki, Grafana Mimir and other telemetry databases from Grafana Labs. If you are already using Grafana Mimir or Pyroscope, you can reuse the storage configuration for setting up Loki. This is an opt-in feature with the Loki 3.4 release. In a future release, Thanos will become the default way of configuring storage and the existing storage clients will be deprecated. @@ -17,34 +17,38 @@ Refer to the [Thanos storage configuration reference](https://grafana.com/docs/l ### Enable the new storage clients -1. Enable Thanos storage clients by setting `use_thanos_objstore` to `true` in the `storage_config` section or by setting the `-use-thanos-objstore` flag to true. When enabled, configuration under `storage_config.object_store` takes effect instead of legacy storage configurations. +1. Enable Thanos storage clients by setting `use_thanos_objstore` to `true` in the `storage_config` section or by setting the `-use-thanos-objstore` flag to true. When enabled, configuration under `storage_config.object_store` takes effect instead of existing storage configurations. ```yaml # Uses the new storage clients for connecting to gcs backend storage_config: - use_thanos_objstore: true # enable the new storage clients - object_store: - gcs: - bucket: "example-bucket" + use_thanos_objstore: true # enable the new storage clients + object_store: + gcs: + bucket_name: "example-bucket" ``` + 1. As an alternative, you can also configure the new clients in the common storage section if you prefer to use the common config section. + ```yaml storage_config: use_thanos_objstore: true # enable the new storage clients common: - storage: - object_store: - gcs: - bucket: "example-bucket" + storage: + object_store: + gcs: + bucket_name: "example-bucket" ``` + 1. Ruler storage should be configured under the `ruler_storage` section when using the new storage clients. + ```yaml storage_config: use_thanos_objstore: true # enable the new storage clients ruler_storage: backend: gcs gcs: - bucket: "example-bucket" + bucket_name: "example-bucket" ``` {{< admonition type="note" >}} @@ -54,19 +58,24 @@ For a complete list of supported parameters, refer to the [Thanos storage config ### GCS Storage Migration -When migrating from the legacy [Google Cloud Storage (GCS)](https://grafana.com/docs/loki//configure/#gcs_storage_config) storage client to the new Thanos-based client, you'll need to update your configuration parameters as follows: +When migrating from the existing [Google Cloud Storage (GCS)](https://grafana.com/docs/loki//configure/#gcs_storage_config) storage client to the new Thanos-based client, you'll need to update your configuration parameters as follows: {{< responsive-table >}} -| Legacy Parameter | New Parameter | Required Changes | -|-----------------|---------------|------------------| -| `bucket_name` | `bucket_name` | No changes required | -| `chunk_buffer_size` | `chunk_size_bytes` | Rename parameter to `chunk_size_bytes` | -| `enable_retries` | `enable_retries` | No changes required | +| Existing Parameter | New Parameter | Required Changes | +|---------------------|-----------------------|--------------------------------------------------------------------------------------------------------------| +| `bucket_name` | `bucket_name` | No changes required | +| `service_account` | `service_account` | No changes required | +| `chunk_buffer_size` | `chunk_buffer_size` | No changes required | +| `enable_retries` | `max_retries` | Replace `enable_retries` (bool) with `max_retries` (int). Set a value > 1 to enable retries, or 1 to disable | +| `request_timeout` | Removed | Remove parameter | +| `enable_opencensus` | Removed | Remove parameter | +| `enable_http2` | Removed | Remove parameter | {{< /responsive-table >}} -**Example configuration migration:** +**Example configuration migration (GCS):** + +Existing configuration: -Legacy configuration: ```yaml storage_config: gcs: @@ -75,73 +84,98 @@ storage_config: enable_retries: true ``` -New configuration: +New configuration (Thanos-based): + ```yaml storage_config: use_thanos_objstore: true -object_store: - gcs: - bucket_name: example-bucket - chunk_size_bytes: 10MB - enable_retries: true + object_store: + gcs: + bucket_name: example-bucket + chunk_buffer_size: 10MB + max_retries: 5 ``` ### Amazon S3 Storage Migration -When migrating from the legacy [Amazon S3](https://grafana.com/docs/loki//configure/#aws_storage_config) storage client to the new Thanos-based client, you'll need to update your configuration parameters as follows: +When migrating from the existing [Amazon S3](https://grafana.com/docs/loki//configure/#aws_storage_config) storage client to the new Thanos-based client, update or remove parameters as follows: {{< responsive-table >}} -| Legacy Parameter | New Parameter | Required Changes | -|-----------------|---------------|------------------| -| `endpoint` | `endpoint` | No changes required | -| `region` | `region` | No changes required | -| `bucket_names` | `bucket_name` | Rename parameter to `bucket_name` | -| `access_key_id` | `access_key_id` | No changes required | -| `secret_access_key` | `secret_access_key` | No changes required | -| `session_token` | `session_token` | No changes required | -| `insecure` | `insecure` | No changes required | -| `disable_dualstack` | `dualstack_enabled` | Rename parameter to `dualstack_enabled` and invert logic | -| `storage_class` | `storage_class` | No changes required | +| Existing Parameter | New Parameter | Required Changes | +|---------------------|-----------------------|------------------------------------------------------------------------------------------------------------------------------------| +| `bucket_names` | `bucket_name` | Rename this parameter. If you previously used multiple buckets, you must consolidate to a single bucket (Thanos supports only one). | +| `endpoint` | `endpoint` | No changes required. | +| `region` | `region` | No changes required. | +| `access_key_id` | `access_key_id` | No changes required. | +| `secret_access_key` | `secret_access_key` | No changes required. | +| `session_token` | `session_token` | No changes required. | +| `insecure` | `insecure` | No changes required. | +| `disable_dualstack` | `dualstack_enabled` | Renamed and inverted. If you had `disable_dualstack: false`, set `dualstack_enabled: true`. | +| `storage_class` | `storage_class` | No changes required. | +| `s3` | Removed | Remove or replace with `endpoint` if you used the URL-based setup. | +| `S3ForcePathStyle` | Removed or replaced | If you need path-based addressing, set `bucket_lookup_type: path` in the new config. Otherwise, remove it. | +| `signature_version` | Removed | Remove parameter. Thanos always uses Signature Version 4 (V4). | +| `http_config` | `http` | Move subfields (such as timeouts, CA file, etc.) into the `http:` block in the Thanos configuration. | +| `sse` | `sse` | Migrate any SSE settings (e.g., `type`, `kms_key_id`) into the `sse:` block in the Thanos configuration. | +| `backoff_config` | `max_retries`| Replace the advanced backoff settings with a single integer (`max_retries`). Set to 1 to disable retries, or a higher value to enable them. | {{< /responsive-table >}} -**Example configuration migration:** +**Example configuration migration (S3):** + +_**Existing**_ -Legacy configuration: ```yaml storage_config: aws: + bucket_names: my-bucket1,my-bucket2 # multiple buckets no longer supported endpoint: s3.amazonaws.com region: us-west-2 - bucket_names: example-bucket access_key_id: example-key secret_access_key: example-secret - insecure: false + signature_version: v4 disable_dualstack: true storage_class: STANDARD + http_config: + timeout: 1m + insecure_skip_verify: false + # ... + backoff_config: + max_retries: 5 + sse: + type: SSE-KMS + kms_key_id: mySSEKey ``` -New configuration: +_**New** (Thanos-based)_ + ```yaml storage_config: use_thanos_objstore: true -object_store: - s3: - endpoint: s3.amazonaws.com - region: us-west-2 - bucket_name: example-bucket - access_key_id: example-key - secret_access_key: example-secret - insecure: false - dualstack_enabled: false - storage_class: STANDARD + object_store: + s3: + bucket_name: my-bucket1 # single bucket + endpoint: s3.amazonaws.com + region: us-west-2 + access_key_id: example-key + secret_access_key: example-secret + dualstack_enabled: false # was disable_dualstack: true + storage_class: STANDARD + max_retries: 5 + http: + insecure_skip_verify: false + sse: + type: SSE-KMS + kms_key_id: mySSEKey ``` +For more advanced configuration options (such as `list_objects_version`, `bucket_lookup_type`, etc.), see the [Thanos S3 configuration reference](https://grafana.com/docs/loki//configure/#thanos_object_store_config). + ### Azure Storage Migration -When migrating from the legacy [Azure](https://grafana.com/docs/loki//configure/#azure_storage_config) storage client to the new Thanos-based client, no changes are required if you are using the following parameters: +When migrating from the existing [Azure](https://grafana.com/docs/loki//configure/#azure_storage_config) storage client to the new Thanos-based client, no changes are required if you are using the following parameters: {{< responsive-table >}} -| Legacy Parameter | New Parameter | Required Changes | +| Existing Parameter | New Parameter | Required Changes | |-----------------|---------------|------------------| | `account_name` | `account_name` | No changes required | | `account_key` | `account_key` | No changes required | @@ -154,3 +188,38 @@ When migrating from the legacy [Azure](https://grafana.com/docs/loki//configure/#filesystem_object_storage_config) +client to the new Thanos-based client, update or remove parameters as follows: + +{{< responsive-table >}} +| Existing Parameter | New Parameter | Required Changes | +|-----------------|---------------|-------------------------------| +| `directory` | `dir` | Rename `directory` to `dir`. | +{{< /responsive-table >}} + +**Example configuration migration (Filesystem):** + +_**Existing** (`FSConfig`)_ + +```yaml +storage_config: + filesystem: + directory: /var/loki/chunks +``` + +_**New** (Thanos-based)_ + +```yaml +storage_config: + use_thanos_objstore: true + object_store: + filesystem: + dir: /var/loki/chunks +``` + +{{< admonition type="note" >}} +For providers not listed here, refer to the [Thanos storage configuration reference](https://grafana.com/docs/loki//configure/#thanos_object_store_config). +{{< /admonition >}} \ No newline at end of file From 86f416d59786ca26ef9e582748a3b6c57226994a Mon Sep 17 00:00:00 2001 From: Ashwanth Goli Date: Fri, 7 Feb 2025 12:09:13 +0530 Subject: [PATCH 5/9] fixup! make corrections to the example configs + update table entries --- .../setup/migrate/migrate-storage-clients/_index.md | 13 +++++-------- 1 file changed, 5 insertions(+), 8 deletions(-) diff --git a/docs/sources/setup/migrate/migrate-storage-clients/_index.md b/docs/sources/setup/migrate/migrate-storage-clients/_index.md index b2b1a1f84c956..1dbb67f6d88da 100644 --- a/docs/sources/setup/migrate/migrate-storage-clients/_index.md +++ b/docs/sources/setup/migrate/migrate-storage-clients/_index.md @@ -8,12 +8,14 @@ weight: Loki release 3.4 introduces new object storage clients based on the [Thanos Object Storage Client Go module](https://github.com/thanos-io/objstore). -One of the reasons for making this change is to have a consistent storage configuration across Grafana Loki, Grafana Mimir and other telemetry databases from Grafana Labs. If you are already using Grafana Mimir or Pyroscope, you can reuse the storage configuration for setting up Loki. +One of the reasons for making this change is to have a consistent storage configuration across Grafana Loki, Mimir and other telemetry databases from Grafana Labs. If you are already using Grafana Mimir or Pyroscope, you can reuse the storage configuration for setting up Loki. This is an opt-in feature with the Loki 3.4 release. In a future release, Thanos will become the default way of configuring storage and the existing storage clients will be deprecated. +{{< admonition type="note" >}} The new storage configuration deviates from the existing format. The following sections describe the changes in detail for each provider. Refer to the [Thanos storage configuration reference](https://grafana.com/docs/loki//configure/#thanos_object_store_config) to view the complete list of supported storage providers and their configuration options. +{{< /admonition >}} ### Enable the new storage clients @@ -50,12 +52,6 @@ Refer to the [Thanos storage configuration reference](https://grafana.com/docs/l gcs: bucket_name: "example-bucket" ``` - -{{< admonition type="note" >}} -Following sections document the steps for migrating the clients. Only some of the commonly used parameters are listed here. -For a complete list of supported parameters, refer to the [Thanos storage configuration reference](https://grafana.com/docs/loki//configure/#thanos_object_store_config) and compare that to what you are using in your setup. -{{< /admonition >}} - ### GCS Storage Migration When migrating from the existing [Google Cloud Storage (GCS)](https://grafana.com/docs/loki//configure/#gcs_storage_config) storage client to the new Thanos-based client, you'll need to update your configuration parameters as follows: @@ -184,6 +180,7 @@ When migrating from the existing [Azure](https://grafana.com/docs/loki/}} If you are using an authentication method other than storage account key or user-assigned managed identity, you'll have to pass the neccessary credetials using environment variables. @@ -191,7 +188,7 @@ For more details, refer to [Azure Identity Client Module for Go](https://pkg.go. ### Filesystem Storage Migration -When migrating from the existing [Filesystem storage](https://grafana.com/docs/loki//configure/#filesystem_object_storage_config) +When migrating from the existing [Filesystem storage](https://grafana.com/docs/loki//configure/#local_storage_config) client to the new Thanos-based client, update or remove parameters as follows: {{< responsive-table >}} From f34fad1650d271bf6270f70d8f4208275fb3ff7b Mon Sep 17 00:00:00 2001 From: Ashwanth Goli Date: Fri, 7 Feb 2025 13:11:11 +0530 Subject: [PATCH 6/9] add examples --- docs/sources/configure/examples/_index.md | 3 +- .../examples/thanos-storage-configs.md | 66 +++++++++++++++++++ 2 files changed, 68 insertions(+), 1 deletion(-) create mode 100644 docs/sources/configure/examples/thanos-storage-configs.md diff --git a/docs/sources/configure/examples/_index.md b/docs/sources/configure/examples/_index.md index 96a00ec3d076a..d4928ef2e113e 100644 --- a/docs/sources/configure/examples/_index.md +++ b/docs/sources/configure/examples/_index.md @@ -10,4 +10,5 @@ weight: 800 The following pages contain examples of how to configure Grafana Loki. - [Configuration snippets and ready-to-use configuration examples]({{< relref "./configuration-examples" >}}). -- [Deploy a query frontend on a existing cluster]({{< relref "./query-frontend" >}}). \ No newline at end of file +- [Deploy a query frontend on a existing cluster]({{< relref "./query-frontend" >}}). +- [Configuration examples for using Thanos-based storage clients](./thanos-storage-configs). diff --git a/docs/sources/configure/examples/thanos-storage-configs.md b/docs/sources/configure/examples/thanos-storage-configs.md new file mode 100644 index 0000000000000..1358f28d627a9 --- /dev/null +++ b/docs/sources/configure/examples/thanos-storage-configs.md @@ -0,0 +1,66 @@ +--- +title: "Configuration examples for using Thanos-based storage clients" +description: "Minimal examples for using Thanos-based S3, GCS, Azure, and filesystem clients in Grafana Loki." +weight: 100 +--- + +# Configuration examples for using Thanos-based storage clients + +Use these examples as a starting point for configuring [Thanos based object storage clients](https://grafana.com/docs/loki//configure/#thanos_object_store_config) in Grafana Loki. + +## GCS example +```yaml +storage_config: + use_thanos_objstore: true + object_store: + gcs: + bucket_name: my-gcs-bucket + + # JSON either from a Google Developers Console client_credentials.json file, + # or a Google Developers service account key. Needs to be valid JSON, not a + # filesystem path. If empty, fallback to Google default logic: + # 1. A JSON file whose path is specified by the GOOGLE_APPLICATION_CREDENTIALS + # environment variable. For workload identity federation, refer to + # https://cloud.google.com/iam/docs/how-to#using-workload-identity-federation + # on how to generate the JSON configuration file for on-prem/non-Google cloud + # platforms. + # 2. A JSON file in a location known to the gcloud command-line tool: + # $HOME/.config/gcloud/application_default_credentials.json. + # 3. On Google Compute Engine it fetches credentials from the metadata server. + service_account: service-account-json +``` + +## S3 example +```yaml +storage_config: + use_thanos_objstore: true + object_store: + s3: + bucket_name: my-s3-bucket + endpoint: s3.us-west-2.amazonaws.com + region: us-west-2 + # You can either declare the access key and secret in the config or + # use environment variables AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY which will be picked up by AWS SDK. + access_key_id: access-key-id + secret_access_key: secret-access-key +``` + +## Azure example +```yaml +storage_config: + use_thanos_objstore: true + object_store: + azure: + account_name: myaccount + account_key: ${SECRET_ACCESS_KEY} # loki expands environment variables + container_name: example-container +``` + +## Filesystem example +```yaml +storage_config: + use_thanos_objstore: true + object_store: + filesystem: + dir: /var/loki/chunks +``` \ No newline at end of file From 6056a4d8e652b1adaf77062dad104d3bb7b3b7fa Mon Sep 17 00:00:00 2001 From: Ashwanth Date: Mon, 10 Feb 2025 10:34:49 +0530 Subject: [PATCH 7/9] Apply suggestions from code review Co-authored-by: J Stickler Signed-off-by: Ashwanth --- .../configure/examples/thanos-storage-configs.md | 3 ++- .../migrate/migrate-storage-clients/_index.md | 14 +++++++------- 2 files changed, 9 insertions(+), 8 deletions(-) diff --git a/docs/sources/configure/examples/thanos-storage-configs.md b/docs/sources/configure/examples/thanos-storage-configs.md index 1358f28d627a9..7ecf2cf392361 100644 --- a/docs/sources/configure/examples/thanos-storage-configs.md +++ b/docs/sources/configure/examples/thanos-storage-configs.md @@ -1,5 +1,6 @@ --- title: "Configuration examples for using Thanos-based storage clients" +menuTitle: Thanos storage examples description: "Minimal examples for using Thanos-based S3, GCS, Azure, and filesystem clients in Grafana Loki." weight: 100 --- @@ -40,7 +41,7 @@ storage_config: endpoint: s3.us-west-2.amazonaws.com region: us-west-2 # You can either declare the access key and secret in the config or - # use environment variables AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY which will be picked up by AWS SDK. + # use environment variables AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY which will be picked up by the AWS SDK. access_key_id: access-key-id secret_access_key: secret-access-key ``` diff --git a/docs/sources/setup/migrate/migrate-storage-clients/_index.md b/docs/sources/setup/migrate/migrate-storage-clients/_index.md index 1dbb67f6d88da..20881ec4aed94 100644 --- a/docs/sources/setup/migrate/migrate-storage-clients/_index.md +++ b/docs/sources/setup/migrate/migrate-storage-clients/_index.md @@ -30,7 +30,7 @@ Refer to the [Thanos storage configuration reference](https://grafana.com/docs/l bucket_name: "example-bucket" ``` -1. As an alternative, you can also configure the new clients in the common storage section if you prefer to use the common config section. +1. As an alternative, you can also configure the new clients in the common `storage` section if you prefer to use the `common` config section. ```yaml storage_config: @@ -70,7 +70,7 @@ When migrating from the existing [Google Cloud Storage (GCS)](https://grafana.co **Example configuration migration (GCS):** -Existing configuration: +_**Existing configuration:**_ ```yaml storage_config: @@ -80,7 +80,7 @@ storage_config: enable_retries: true ``` -New configuration (Thanos-based): +_**New configuration**_ (Thanos-based): ```yaml storage_config: @@ -118,7 +118,7 @@ When migrating from the existing [Amazon S3](https://grafana.com/docs/loki/ Date: Mon, 10 Feb 2025 15:11:20 +0530 Subject: [PATCH 8/9] add a point about storage_prefix --- .../setup/migrate/migrate-storage-clients/_index.md | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/docs/sources/setup/migrate/migrate-storage-clients/_index.md b/docs/sources/setup/migrate/migrate-storage-clients/_index.md index 20881ec4aed94..fad6d434eb1b6 100644 --- a/docs/sources/setup/migrate/migrate-storage-clients/_index.md +++ b/docs/sources/setup/migrate/migrate-storage-clients/_index.md @@ -52,6 +52,17 @@ Refer to the [Thanos storage configuration reference](https://grafana.com/docs/l gcs: bucket_name: "example-bucket" ``` + +1. If you are using `store.object-prefix` flag or the corresponding `object_prefix` YAML setting, you'll need to update your configuration to use the new `object_store.storage-prefix` flag or the corresponding `storage_prefix` YAML setting. + + ```yaml + # Example configuration to prefix all objects with "prefix" + storage_config: + use_thanos_objstore: true # enable the new storage clients + object_store: + storage_prefix: "prefix" + ``` + ### GCS Storage Migration When migrating from the existing [Google Cloud Storage (GCS)](https://grafana.com/docs/loki//configure/#gcs_storage_config) storage client to the new Thanos-based client, you'll need to update your configuration parameters as follows: From 13641cf1aac7714c1e6e1946dc66faf857b539fc Mon Sep 17 00:00:00 2001 From: Ashwanth Goli Date: Tue, 11 Feb 2025 18:39:34 +0530 Subject: [PATCH 9/9] include service_account inline --- .../configure/examples/thanos-storage-configs.md | 16 ++++++++++++++-- 1 file changed, 14 insertions(+), 2 deletions(-) diff --git a/docs/sources/configure/examples/thanos-storage-configs.md b/docs/sources/configure/examples/thanos-storage-configs.md index 7ecf2cf392361..b24ba9698b7ae 100644 --- a/docs/sources/configure/examples/thanos-storage-configs.md +++ b/docs/sources/configure/examples/thanos-storage-configs.md @@ -28,7 +28,19 @@ storage_config: # 2. A JSON file in a location known to the gcloud command-line tool: # $HOME/.config/gcloud/application_default_credentials.json. # 3. On Google Compute Engine it fetches credentials from the metadata server. - service_account: service-account-json + service_account: |- + { + "type": "service_account", + "project_id": "project", + "private_key_id": "abcdefghijklmnopqrstuvwxyz12345678906666", + "private_key": "-----BEGIN PRIVATE KEY-----\...\n-----END PRIVATE KEY-----\n", + "client_email": "project@example.iam.gserviceaccount.com", + "client_id": "123456789012345678901", + "auth_uri": "https://accounts.google.com/o/oauth2/auth", + "token_uri": "https://oauth2.googleapis.com/token", + "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs", + "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/project@example.iam.gserviceaccount.com" + } ``` ## S3 example @@ -64,4 +76,4 @@ storage_config: object_store: filesystem: dir: /var/loki/chunks -``` \ No newline at end of file +```