[Profiling] Improve docs on how to manage storage

docs/en/observability/index.asciidoc

@@ -141,11 +141,14 @@ include::universal-profiling.asciidoc[leveloffset=+1]
 
 include::profiling-get-started.asciidoc[leveloffset=+2]
 
+include::profiling-manage-storage.asciidoc[leveloffset=+2]
+include::profiling-index-lifecycle-management.asciidoc[leveloffset=+3]
+include::profiling-probabilistic-profiling.asciidoc[leveloffset=+3]
+
 include::profiling-advanced-configuration.asciidoc[leveloffset=+2]
 include::profiling-tag-data-query.asciidoc[leveloffset=+3]
 include::profiling-add-symbols.asciidoc[leveloffset=+3]
 include::profiling-use-a-proxy.asciidoc[leveloffset=+3]
-include::profiling-probabilistic-profiling.asciidoc[leveloffset=+3]
 include::profiling-no-kernel-version-check.asciidoc[leveloffset=+3]
 
 include::profiling-upgrade.asciidoc[leveloffset=+2]

docs/en/observability/profiling-advanced-configuration.asciidoc

@@ -7,8 +7,6 @@ See the following sections for more information:
 * <<profiling-tag-data-query, Tag data for querying>>: Tag data collected by the host-agent into multiple logical groups so they can be queried in Kibana.
 * <<profiling-add-symbols, Add symbols for native frames>>: Push symbols to your cluster so you can see function names and line numbers in traces of applications written in programming languages that compile to native code (C, C++, Rust, Go, etc.).
 * <<profiling-use-a-proxy,Use a proxy>>:  Set up an HTTP proxy if your infrastructure host-agent installation needs one to reach {ecloud}.
-* <<profiling-probabilistic-profiling, Configure probabilistic profiling>>: Configure the Universal Profiling agent to run in probabilistic profiling mode.
-
-* <<profiling-no-kernel-version-check, Override kernel version check >>: Configure the Universal Profiling agent to bypass the kernel version compatibility check. 
+* <<profiling-no-kernel-version-check, Override kernel version check >>: Configure the Universal Profiling agent to bypass the kernel version compatibility check.
 
 

docs/en/observability/profiling-index-lifecycle-management.asciidoc

@@ -0,0 +1,157 @@
+[[profiling-index-lifecycle-management]]
+= Universal Profiling index life cycle management
+
+++++
+<titleabbrev>Index lifecycle management</titleabbrev>
+++++
+
+
+Index lifecycle policies allow you to automate the lifecycle of your profiling indices as they grow and age. A default policy is applied, but you can customize it based on your business needs.
+
+[discrete]
+[[profiling-ilm-default-policy]]
+== Default policy
+
+The default Universal Profiling index lifecycle policy includes the following rollover and delete definitions:
+
+
+* **Rollover**: Rollover prevents a single index from growing too large and optimizes indexing and search performance. After an age or size metric threshold is met, a new index is created and all subsequent updates are written to the new index.
+
+* **Delete**: The delete phase permanently removes the index after a time threshold is met.
+
+The following table lists the default thresholds for rollover and delete:
+
+[cols="1,1,1",options="header"]
+|===
+|Rollover
+|Warm tier
+|Delete
+
+| after 30 days or 50 GB
+| after 30 days
+| after 60 days
+
+
+|===
+
+NOTE: The {ref}/ilm-rollover.html#_rollover_condition_blocks_phase_transition[rollover condition blocks phase transitions] which means that indices are kept 30 days *after* rollover on the hot tier.
+
+To view the Universal Profiling index lifecycle policy in {kib}, navigate to *{stack-manage-app}* → *Index Lifecycle Management*, and search for `profiling`.
+
+
+TIP: Default {ilm-init} policies can change between minor versions. This is not considered a breaking change as index management should continually improve and adapt to new features.
+
+[discrete]
+[[profiling-ilm-custom-policy]]
+== Configure a custom index lifecycle policy
+
+[discrete]
+[[profiling-ilm-custom-policy-create-policy]]
+=== Step 1: Create an index lifecycle policy
+
+. Navigate to **{stack-manage-app}** → **Index Lifecycle Policies**.
+
+. Click **Create policy**.
+. Name your new policy, for example `custom-profiling-policy`. 
+. Customize the policy to your liking.
+. Click **Save policy**.
+
+TIP: See {ref}/index-lifecycle-management.html[Manage the index lifecycle] to learn more about {ilm-init} policies.
+
+[discrete]
+[[profiling-ilm-custom-policy-apply-policy]]
+=== Step 2: Apply the index lifecycle policy
+
+To apply your new index lifecycle policy for Universal Profiling, create a new component template named `profiling-ilm@custom`.
+
+NOTE: To apply a custom {ilm-init} policy, you must name the component template `profiling-ilm@custom`. Other names are not supported.
+
+
+. From the *Index Management* page, select the **Component Template** tab and click **Create component template**.
+
+. Enter `profiling-ilm@custom` as the name and click **Next**.
+
+. In **Index settings**, set the {ilm-init} policy name created in the previous step:
++
+[source,json]
+----
+{
+  "lifecycle": {
+    "name": "custom-profiling-policy"
+  }
+}
+----
+. Continue to the **Review** step, and select the *Request* tab. Your request should look similar to the following image.
+
+
+If it does, click **Create component template**.
++
+[role="screenshot"]
+image::images/profiling-create-component-template.png[Create component template]
+
+[discrete]
+[[profiling-ilm-custom-policy-rollover]]
+=== Step 3: Rollover indices
+
+To confirm that Universal Profiling is now using the new index template and {ilm-init} policy, navigate to **{dev-tools-app}** and run the following:
+
+[source,bash]
+----
+GET _ilm/policy/custom-profiling-policy <1>
+----
+<1> The name of the custom {ilm-init} policy chosen in <<profiling-ilm-custom-policy-create-policy,Step 1>>.
+
+
+If the custom policy is already applied, the result should include the following:
+
+[source,json]
+----
+{
+    "in_use_by": {
+      "indices": [
+        ...
+      ],
+      "data_streams": [
+        ...
+        "profiling-events-all",
+        ...
+      ],
+      "composable_templates": [
+        "profiling-stackframes",
+        "profiling-symbols-global",
+        "profiling-metrics",
+        "profiling-stacktraces",
+        "profiling-executables",
+        "profiling-hosts",
+        "profiling-events"
+      ]
+    }
+}
+----
+
+If the result is empty, the custom {ilm-init} policy is not yet in use. New {ilm-init} policies only take effect when new indices are created, so either wait for a rollover to occur (usually after 30 days or when the index size reaches 50 GB), or force a rollover using the {ref}/indices-rollover-index.html[{es} rollover API]:
+
+
+[source,bash]
+----
+POST /profiling-events-5pow01/_rollover/
+POST /profiling-events-5pow02/_rollover/
+POST /profiling-events-5pow03/_rollover/
+POST /profiling-events-5pow04/_rollover/
+POST /profiling-events-5pow05/_rollover/
+POST /profiling-events-5pow06/_rollover/
+POST /profiling-events-5pow07/_rollover/
+POST /profiling-events-5pow08/_rollover/
+POST /profiling-events-5pow09/_rollover/
+POST /profiling-events-5pow10/_rollover/
+POST /profiling-events-5pow11/_rollover/
+POST /profiling-events-all/_rollover/
+POST /profiling-executables/_rollover/
+POST /profiling-hosts/_rollover/
+POST /profiling-metrics/_rollover/
+POST /profiling-stackframes/_rollover/
+POST /profiling-stacktraces/_rollover/
+POST /profiling-symbols-global/_rollover/
+----
+
+After the rollover, the custom {ilm-init} policy will be applied to new indices and data streams.

docs/en/observability/profiling-manage-storage.asciidoc

@@ -0,0 +1,11 @@
+[[profiling-manage-storage]]
+= Manage data storage
+
+
+Universal Profiling provides the following ways to manage how your data is stored. 
+
+* <<profiling-index-lifecycle-management, Index lifecycle management>> automatically manages your indices according to age or size metric thresholds. Universal Profiling ships with a default index lifecycle policy, but you can create a custom policy to meet your requirements. 
+* <<profiling-probabilistic-profiling, Probabilistic profiling>> mode uses representative samples of profiling data to reduce storage needs even further.
+
+
+