[Security Solution] Refactor prebuilt rule customization test plans, …

…part 1 (#211300) **Epic:** #174168 **Partially addresses:** #202068, #202078, #202079 ## Summary We're cleaning up and refactoring our existing test plans for prebuilt rule customization, upgrade, and export/import workflows. Specifically, in this PR we're consistently: - Changing the file names and test plan titles. - Adding or expanding test plan summaries. - Regenerating tables of contents using the [Markdown All in One](https://marketplace.visualstudio.com/items?itemName=yzhang.markdown-all-in-one) VS Code extension. No "functional" changes have been made to any test plans, such as adding, removing, or updating any scenarios. This refactoring prepares the test plans for being "functionally" changed and improved in follow-up PRs. For example, we're going to cover the logic described in #210358 and address any other gaps in coverage.
elastic · Feb 14, 2025 · 9600de4 · 9600de4
1 parent 819fd7a
commit 9600de4
Show file tree

Hide file tree

Showing 10 changed files with 199 additions and 46 deletions.
diff --git a/...onse/prebuilt_rules/rule_customization.md → ...uilt_rules/prebuilt_rule_customization.md b/...onse/prebuilt_rules/rule_customization.md → ...uilt_rules/prebuilt_rule_customization.md
@@ -1,8 +1,54 @@
-# Prebuilt Rule Customization Workflows
-
-This is a test plan for rule customization workflows specifically related to prebuilt rules
-
-Status: `in progress`.
+# Test plan: customizing prebuilt rules <!-- omit from toc -->
+
+**Status**: `in progress`, matches [Milestone 3](https://github.com/elastic/kibana/issues/174168).
+
+## Summary <!-- omit from toc -->
+
+This is a test plan for the workflows of customizing prebuilt rules via:
+
+- editing single rules one-by-one on the Rule Editing page:
+  - initiated from the Rule Details page
+  - initiated from the Rule Management page
+- editing multiple rules in bulk on the Rule Management page via bulk actions, such as:
+  - bulk adding or removing index patterns
+  - bulk updating rule schedule
+
+as well as un-customizing prebuilt rules by reverting rule parameters back to their original values.
+
+## Table of contents <!-- omit from toc -->
+
+<!--
+Please use the "Markdown All in One" VS Code extension to keep the TOC in sync with the text:
+https://marketplace.visualstudio.com/items?itemName=yzhang.markdown-all-in-one
+-->
+
+- [Useful information](#useful-information)
+  - [Tickets](#tickets)
+  - [Terminology](#terminology)
+  - [Assumptions](#assumptions)
+- [Scenarios](#scenarios)
+  - [Editing prebuilt rules](#editing-prebuilt-rules)
+    - [**Scenario: User can edit a non-customized prebuilt rule from the rule edit page**](#scenario-user-can-edit-a-non-customized-prebuilt-rule-from-the-rule-edit-page)
+    - [**Scenario: User can edit a customized prebuilt rule from the rule edit page**](#scenario-user-can-edit-a-customized-prebuilt-rule-from-the-rule-edit-page)
+    - [**Scenario: User can navigate to rule editing page from the rule details page**](#scenario-user-can-navigate-to-rule-editing-page-from-the-rule-details-page)
+    - [**Scenario: User can navigate to rule editing page from the rule management page**](#scenario-user-can-navigate-to-rule-editing-page-from-the-rule-management-page)
+    - [**Scenario: User can bulk edit prebuilt rules from rules management page**](#scenario-user-can-bulk-edit-prebuilt-rules-from-rules-management-page)
+  - [Detecting rule customizations](#detecting-rule-customizations)
+    - [**Scenario: is\_customized is set to true when user edits a customizable rule field**](#scenario-is_customized-is-set-to-true-when-user-edits-a-customizable-rule-field)
+    - [**Scenario: is\_customized calculation is not affected by specific fields**](#scenario-is_customized-calculation-is-not-affected-by-specific-fields)
+    - [**Scenario: User cannot change non-customizable rule fields on prebuilt rules**](#scenario-user-cannot-change-non-customizable-rule-fields-on-prebuilt-rules)
+    - [**Scenario: User can revert a customized prebuilt rule to its original state**](#scenario-user-can-revert-a-customized-prebuilt-rule-to-its-original-state)
+  - [Calculating the Modified badge in the UI](#calculating-the-modified-badge-in-the-ui)
+    - [**Scenario: Modified badge should appear on the rule details page when prebuilt rule is customized**](#scenario-modified-badge-should-appear-on-the-rule-details-page-when-prebuilt-rule-is-customized)
+    - [**Scenario: Modified badge should not appear on the rule details page when prebuilt rule isn't customized**](#scenario-modified-badge-should-not-appear-on-the-rule-details-page-when-prebuilt-rule-isnt-customized)
+    - [**Scenario: Modified badge should not appear on a custom rule's rule details page**](#scenario-modified-badge-should-not-appear-on-a-custom-rules-rule-details-page)
+    - [**Scenario: Modified badge should appear on the rule management table when prebuilt rule is modified**](#scenario-modified-badge-should-appear-on-the-rule-management-table-when-prebuilt-rule-is-modified)
+    - [**Scenario: Modified badge should not appear on the rule management table when prebuilt rule isn't customized**](#scenario-modified-badge-should-not-appear-on-the-rule-management-table-when-prebuilt-rule-isnt-customized)
+    - [**Scenario: Modified badge should not appear on the rule management table when row is a custom rule**](#scenario-modified-badge-should-not-appear-on-the-rule-management-table-when-row-is-a-custom-rule)
+    - [**Scenario: Modified badge should appear on the rule updates table when prebuilt rule is customized**](#scenario-modified-badge-should-appear-on-the-rule-updates-table-when-prebuilt-rule-is-customized)
+    - [**Scenario: Modified badge should not appear on the rule updates table when prebuilt rule isn't customized**](#scenario-modified-badge-should-not-appear-on-the-rule-updates-table-when-prebuilt-rule-isnt-customized)
+    - [**Scenario: User should be able to see only customized rules in the rule updates table**](#scenario-user-should-be-able-to-see-only-customized-rules-in-the-rule-updates-table)
+    - [**Scenario: User should be able to filter by non-customized rules on the rule updates table**](#scenario-user-should-be-able-to-filter-by-non-customized-rules-on-the-rule-updates-table)
 
 ## Useful information
 

diff --git a/...tion_response/prebuilt_rules/exporting.md → ...se/prebuilt_rules/prebuilt_rule_export.md b/...tion_response/prebuilt_rules/exporting.md → ...se/prebuilt_rules/prebuilt_rule_export.md
@@ -1,8 +1,42 @@
-# Detection Rule Export
+# Test plan: exporting prebuilt rules <!-- omit from toc -->
 
-This is a test plan for the exporting of prebuilt and custom rules. This feature is an aspect of `Milestone 3` of the [Rule Immutability/Customization](https://github.com/elastic/security-team/issues/1974) epic.
+**Status**: `in progress`, matches [Milestone 3](https://github.com/elastic/kibana/issues/174168).
 
-Status: `in progress`.
+## Summary <!-- omit from toc -->
+
+This is a test plan for the workflows of:
+
+- exporting single prebuilt rules from the Rule Details page
+- exporting single prebuilt rules one-by-one from the Rule Management page
+- exporting multiple prebuilt rules in bulk from the Rule Management page
+- exporting a mixture of prebuilt and custom rules from the Rule Management page
+
+where each prebuilt rule:
+
+- can be an original (non-customized) prebuilt rule from Elastic, or
+- can be a prebuilt rule customized by the user.
+
+## Table of contents <!-- omit from toc -->
+
+<!--
+Please use the "Markdown All in One" VS Code extension to keep the TOC in sync with the text:
+https://marketplace.visualstudio.com/items?itemName=yzhang.markdown-all-in-one
+-->
+
+- [Useful information](#useful-information)
+  - [Tickets](#tickets)
+  - [Terminology](#terminology)
+- [Scenarios](#scenarios)
+  - [Core Functionality](#core-functionality)
+    - [Scenario: Exporting prebuilt rule individually from rule details page](#scenario-exporting-prebuilt-rule-individually-from-rule-details-page)
+    - [Scenario: Exporting custom rule individually from rule details page](#scenario-exporting-custom-rule-individually-from-rule-details-page)
+    - [Scenario: Exporting prebuilt rule individually from rules management table](#scenario-exporting-prebuilt-rule-individually-from-rules-management-table)
+    - [Scenario: Exporting custom rule individually from rules management table](#scenario-exporting-custom-rule-individually-from-rules-management-table)
+    - [Scenario: Exporting prebuilt rules in bulk](#scenario-exporting-prebuilt-rules-in-bulk)
+    - [Scenario: Exporting custom rules in bulk](#scenario-exporting-custom-rules-in-bulk)
+    - [Scenario: Exporting both prebuilt and custom rules in bulk](#scenario-exporting-both-prebuilt-and-custom-rules-in-bulk)
+  - [Error Handling](#error-handling)
+    - [Scenario: Exporting beyond the export limit](#scenario-exporting-beyond-the-export-limit)
 
 ## Useful information
 

diff --git a/...tion_response/prebuilt_rules/importing.md → ...se/prebuilt_rules/prebuilt_rule_import.md b/...tion_response/prebuilt_rules/importing.md → ...se/prebuilt_rules/prebuilt_rule_import.md
@@ -1,8 +1,43 @@
-# Detections Rule Import
-
-This is a test plan for the importing of prebuilt and custom rules. This feature is an aspect of `Milestone 3` of the [Rule Immutability/Customization](https://github.com/elastic/security-team/issues/1974) epic.
-
-Status: `in progress`.
+# Test plan: importing prebuilt rules <!-- omit from toc -->
+
+**Status**: `in progress`, matches [Milestone 3](https://github.com/elastic/kibana/issues/174168).
+
+## Summary <!-- omit from toc -->
+
+This is a test plan for the workflows of:
+
+- importing prebuilt non-customized rules
+- importing prebuilt customized rules
+- importing any mixture of prebuilt and custom rules
+
+from the Rule Management page.
+
+## Table of contents <!-- omit from toc -->
+
+<!--
+Please use the "Markdown All in One" VS Code extension to keep the TOC in sync with the text:
+https://marketplace.visualstudio.com/items?itemName=yzhang.markdown-all-in-one
+-->
+
+- [Useful information](#useful-information)
+  - [Tickets](#tickets)
+  - [Terminology](#terminology)
+- [Scenarios](#scenarios)
+  - [Core Functionality](#core-functionality)
+    - [Scenario: Importing an unmodified prebuilt rule with a matching rule\_id and version](#scenario-importing-an-unmodified-prebuilt-rule-with-a-matching-rule_id-and-version)
+    - [Scenario: Importing a customized prebuilt rule with a matching rule\_id and version](#scenario-importing-a-customized-prebuilt-rule-with-a-matching-rule_id-and-version)
+    - [Scenario: Importing a custom rule with a matching rule\_id and version](#scenario-importing-a-custom-rule-with-a-matching-rule_id-and-version)
+    - [Scenario: Importing a prebuilt rule with a matching rule\_id but no matching version](#scenario-importing-a-prebuilt-rule-with-a-matching-rule_id-but-no-matching-version)
+    - [Scenario: Importing a prebuilt rule with a non-existent rule\_id](#scenario-importing-a-prebuilt-rule-with-a-non-existent-rule_id)
+    - [Scenario: Importing a prebuilt rule without a rule\_id field](#scenario-importing-a-prebuilt-rule-without-a-rule_id-field)
+    - [Scenario: Importing a prebuilt rule with a matching rule\_id but missing a version field](#scenario-importing-a-prebuilt-rule-with-a-matching-rule_id-but-missing-a-version-field)
+    - [Scenario: Importing an existing custom rule missing a version field](#scenario-importing-an-existing-custom-rule-missing-a-version-field)
+    - [Scenario: Importing a new custom rule missing a version field](#scenario-importing-a-new-custom-rule-missing-a-version-field)
+    - [Scenario: Importing a rule with overwrite flag set to true](#scenario-importing-a-rule-with-overwrite-flag-set-to-true)
+    - [Scenario: Importing a rule with overwrite flag set to false](#scenario-importing-a-rule-with-overwrite-flag-set-to-false)
+    - [Scenario: Importing both custom and prebuilt rules](#scenario-importing-both-custom-and-prebuilt-rules)
+    - [Scenario: Importing prebuilt rules when the rules package is not installed](#scenario-importing-prebuilt-rules-when-the-rules-package-is-not-installed)
+    - [Scenario: User imports a custom rule before a prebuilt rule asset is created with the same rule\_id](#scenario-user-imports-a-custom-rule-before-a-prebuilt-rule-asset-is-created-with-the-same-rule_id)
 
 ## Useful information
 

diff --git a/...n_response/prebuilt_rules/installation.md → ...built_rules/prebuilt_rule_installation.md b/...n_response/prebuilt_rules/installation.md → ...built_rules/prebuilt_rule_installation.md
@@ -1,10 +1,22 @@
-# Installation of Prebuilt Rules
+# Test plan: installing prebuilt rules <!-- omit from toc -->
 
-This is a test plan for the workflows of installing prebuilt rules.
+**Status**: `in progress`, matches [Milestone 3](https://github.com/elastic/kibana/issues/174168).
 
-Status: `in progress`. The current test plan matches [Rule Immutability/Customization Milestone 3 epic](https://github.com/elastic/kibana/issues/174168).
+## Summary <!-- omit from toc -->
 
-## Table of Contents
+This is a test plan for the workflows of:
+
+- installing single prebuilt rules one-by-one
+- installing multiple prebuilt rules in bulk
+
+from the Rule Installation page.
+
+## Table of contents <!-- omit from toc -->
+
+<!--
+Please use the "Markdown All in One" VS Code extension to keep the TOC in sync with the text:
+https://marketplace.visualstudio.com/items?itemName=yzhang.markdown-all-in-one
+-->
 
 - [Useful information](#useful-information)
   - [Tickets](#tickets)
@@ -13,7 +25,7 @@ Status: `in progress`. The current test plan matches [Rule Immutability/Customiz
   - [Non-functional requirements](#non-functional-requirements)
   - [Functional requirements](#functional-requirements)
 - [Scenarios](#scenarios)
-  - [Rule installation notifications on the Rule Management page](#rule-installation-and-upgrade-notifications-on-the-rule-management-page)
+  - [Rule installation notifications on the Rule Management page](#rule-installation-notifications-on-the-rule-management-page)
     - [**Scenario: User is NOT notified when no prebuilt rules are installed and there are no prebuilt rules assets**](#scenario-user-is-not-notified-when-no-prebuilt-rules-are-installed-and-there-are-no-prebuilt-rules-assets)
     - [**Scenario: User is NOT notified when all prebuilt rules are installed and up to date**](#scenario-user-is-not-notified-when-all-prebuilt-rules-are-installed-and-up-to-date)
     - [**Scenario: User is notified when no prebuilt rules are installed and there are rules available to install**](#scenario-user-is-notified-when-no-prebuilt-rules-are-installed-and-there-are-rules-available-to-install)
@@ -28,17 +40,17 @@ Status: `in progress`. The current test plan matches [Rule Immutability/Customiz
     - [**Scenario: User can preview rules available for installation**](#scenario-user-can-preview-rules-available-for-installation)
     - [**Scenario: User can install a rule using the rule preview**](#scenario-user-can-install-a-rule-using-the-rule-preview)
     - [**Scenario: User can see correct rule information in preview before installing**](#scenario-user-can-see-correct-rule-information-in-preview-before-installing)
-    - [**Scenario: Tabs and sections without content should be hidden in preview before installing**](#scenario-tabs-and-sections-without-content-should-be-hidden-in-preview-before-installing)
+    - [**Scenario: Optional tabs and sections without content should be hidden in preview before installing**](#scenario-optional-tabs-and-sections-without-content-should-be-hidden-in-preview-before-installing)
   - [Rule installation workflow: filtering, sorting, pagination](#rule-installation-workflow-filtering-sorting-pagination)
   - [Rule installation workflow: misc cases](#rule-installation-workflow-misc-cases)
     - [**Scenario: User opening the Add Rules page sees a loading skeleton until the package installation is completed**](#scenario-user-opening-the-add-rules-page-sees-a-loading-skeleton-until-the-package-installation-is-completed)
     - [**Scenario: User can navigate from the Add Rules page to the Rule Management page via breadcrumbs**](#scenario-user-can-navigate-from-the-add-rules-page-to-the-rule-management-page-via-breadcrumbs)
-  - [Rule installation and upgrade via the Prebuilt rules API](#rule-installation-and-upgrade-via-the-prebuilt-rules-api)
+  - [Rule installation via the Prebuilt rules API](#rule-installation-via-the-prebuilt-rules-api)
     - [**Scenario: API can install all prebuilt rules**](#scenario-api-can-install-all-prebuilt-rules)
     - [**Scenario: API can install prebuilt rules that are not yet installed**](#scenario-api-can-install-prebuilt-rules-that-are-not-yet-installed)
-    - [**Scenario: API does not install prebuilt rules if they are up to date**](#scenario-api-does-not-installupgrade-prebuilt-rules-if-they-are-up-to-date)
+    - [**Scenario: API does not install prebuilt rules if they are up to date**](#scenario-api-does-not-install-prebuilt-rules-if-they-are-up-to-date)
   - [Error handling](#error-handling)
-    - [**Scenario: Error is handled when any operation on prebuilt rules fails**](#scenario-error-is-handled-when-any-operation-on-prebuilt-rules-fails)
+    - [**Scenario: Error is handled when any installation operation on prebuilt rules fails**](#scenario-error-is-handled-when-any-installation-operation-on-prebuilt-rules-fails)
   - [Authorization / RBAC](#authorization--rbac)
     - [**Scenario: User with read privileges on Security Solution cannot install prebuilt rules**](#scenario-user-with-read-privileges-on-security-solution-cannot-install-prebuilt-rules)
 

diff --git a/...ebuilt_rules/upgrade_review_algorithms.md → .../prebuilt_rule_upgrade_diff_algorithms.md b/...ebuilt_rules/upgrade_review_algorithms.md → .../prebuilt_rule_upgrade_diff_algorithms.md
@@ -1,19 +1,25 @@
-# Diff Algorithms for `upgrade/_review` Endpoint
+# Test plan: diff algorithms for upgrading prebuilt rules <!-- omit from toc -->
 
-This is a test plan for the `upgrade/_review` endpoint diff algorithms that are a part of the larger prebuilt rules customization feature. These algorithms determine what fields get returned when a user makes an API request to review changes as a part of the rule update process and determine what version of those fields should be displayed by the UI.
+**Status**: `in progress`, matches [Milestone 3](https://github.com/elastic/kibana/issues/174168).
 
-Status: `in progress`.
+## Summary <!-- omit from toc -->
 
-## Table of Contents
+This is a test plan for the diff algorithms used in the workflows of upgrading prebuilt rules and specifically in the `upgrade/_review` endpoint.
+
+These algorithms determine what fields get returned when a user makes an API request to review changes as a part of the rule upgrade process and determine what version of those fields should be displayed by the UI.
+
+## Table of contents <!-- omit from toc -->
+
+<!--
+Please use the "Markdown All in One" VS Code extension to keep the TOC in sync with the text:
+https://marketplace.visualstudio.com/items?itemName=yzhang.markdown-all-in-one
+-->
 
 - [Useful information](#useful-information)
   - [Tickets](#tickets)
   - [Terminology](#terminology)
   - [Assumptions](#assumptions)
-  - [Non-functional requirements](#non-functional-requirements)
-  - [Functional requirements](#functional-requirements)
 - [Scenarios](#scenarios)
-
   - [Rule field doesn't have an update and has no custom value - `AAA`](#rule-field-doesnt-have-an-update-and-has-no-custom-value---aaa)
     - [**Scenario: `AAA` - Rule field is any type**](#scenario-aaa---rule-field-is-any-type)
   - [Rule field doesn't have an update but has a custom value - `ABA`](#rule-field-doesnt-have-an-update-but-has-a-custom-value---aba)

diff --git a/...se/prebuilt_rules/upgrade_with_preview.md → ...les/prebuilt_rule_upgrade_with_preview.md b/...se/prebuilt_rules/upgrade_with_preview.md → ...les/prebuilt_rule_upgrade_with_preview.md
@@ -1,10 +1,16 @@
-# Upgrading prebuilt rules one-by-one with preview <!-- omit from toc -->
+# Test plan: upgrading prebuilt rules one-by-one with preview <!-- omit from toc -->
 
-This is a test plan for the workflow of upgrading prebuilt rules one-by-one with previewing the incoming changes and user customizations in the Rule Upgrade flyout.
+**Status**: `in progress`, matches [Milestone 3](https://github.com/elastic/kibana/issues/174168).
 
-Status: `in progress`. The current test plan matches [Milestone 3](https://github.com/elastic/kibana/issues/174168).
+## Summary <!-- omit from toc -->
 
-## Table of Contents <!-- omit from toc -->
+This is a test plan for the workflow of:
+
+- upgrading single prebuilt rules one-by-one
+
+from the Rule Upgrade table with previewing incoming updates from Elastic and user customizations in the Rule Upgrade flyout.
+
+## Table of contents <!-- omit from toc -->
 
 <!--
 Please use the "Markdown All in One" VS Code extension to keep the TOC in sync with the text:

diff --git a/...ection_response/prebuilt_rules/upgrade.md → .../prebuilt_rule_upgrade_without_preview.md b/...ection_response/prebuilt_rules/upgrade.md → .../prebuilt_rule_upgrade_without_preview.md
@@ -1,13 +1,20 @@
-# Upgrade of Prebuilt Rules <!-- omit from toc -->
+# Test plan: upgrading prebuilt rules one-by-one or in bulk without preview <!-- omit from toc -->
 
-This is a test plan for the workflow of upgrading prebuilt rules.
+**Status**: `in progress`, matches [Milestone 3](https://github.com/elastic/kibana/issues/174168).
 
-Status: `in progress`. The current test plan matches [Milestone 3](https://github.com/elastic/kibana/issues/174168).
+## Summary <!-- omit from toc -->
 
-## Table of Contents <!-- omit from toc -->
+This is a test plan for the workflows of:
+
+- upgrading single prebuilt rules one-by-one
+- upgrading multiple prebuilt rules in bulk
+
+from the Rule Upgrade table without previewing incoming updates from Elastic and user customizations in the Rule Upgrade flyout.
+
+## Table of contents <!-- omit from toc -->
 
 <!--
-Use Markdown All in One in VS Code to keep the TOC in sync with the text:
+Please use the "Markdown All in One" VS Code extension to keep the TOC in sync with the text:
 https://marketplace.visualstudio.com/items?itemName=yzhang.markdown-all-in-one
 -->