[DaC] [FR] Ndjson support for action connectors (#3955)

* [New Rule] AWS IAM CompromisedKeyQuarantine Policy Attached to User (#3910) * [New Rule] AWS IAM CompromisedKeyQuarantine Policy Attached to User * increased severity score Co-authored-by: Ruben Groenewoud <[email protected]> --------- Co-authored-by: Ruben Groenewoud <[email protected]> * [Rule Tuning] System Binary Moved or Copied (#3933) * [Rule Tuning] Sensitive Registry Hive Access via RegBack (#3947) Co-authored-by: Mika Ayenson <[email protected]> * [New Rule] Potential Relay Attack against a Domain Controller (#3928) * [New Rule] Potential Relay Attack against a Domain Controller * Update credential_access_dollar_account_relay.toml * Move to the correct folder * [Rule Tuning] AWS S3 Object Versioning Suspended (#3953) * [Tuning] Executable Bit Set for Potential Persistence Script (#3929) * [Rule Tuning] Microsoft IIS Service Account Password Dumped (#3935) * [Rule Tuning] Accepted Default Telnet Port Connection (#3954) Co-authored-by: Mika Ayenson <[email protected]> * ndjson support for action connectors * Add kibana API support for actions * Minor typo * Add actions connector support * update rule formatter * Fix typo * [New Rule] Outlook Home Page Registry Modification (#3946) * Fix typos * Update docs and generated config * Update all_versions * Fix comments --------- Co-authored-by: Isai <[email protected]> Co-authored-by: Ruben Groenewoud <[email protected]> Co-authored-by: Jonhnathan <[email protected]> Co-authored-by: Mika Ayenson <[email protected]>
elastic · Aug 5, 2024 · 90643bf · 90643bf
1 parent 66986ef
commit 90643bf
Show file tree

Hide file tree

Showing 14 changed files with 367 additions and 25 deletions.
diff --git a/CLI.md b/CLI.md
@@ -85,12 +85,18 @@ Usage: detection_rules import-rules-to-repo [OPTIONS] [INPUT_FILE]...
   Import rules from json, toml, yaml, or Kibana exported rule file(s).
 
 Options:
+  -ac, --action-connector-import  Include action connectors in export
   -e, --exceptions-import         Include exceptions in export
   --required-only                 Only prompt for required fields
   -d, --directory DIRECTORY       Load files from a directory
   -s, --save-directory DIRECTORY  Save imported rules to a directory
   -se, --exceptions-directory DIRECTORY
                                   Save imported exceptions to a directory
+  -sa, --action-connectors-directory DIRECTORY
+                                  Save imported actions to a directory
+  -ske, --skip-errors             Skip rule import errors
+  -da, --default-author TEXT      Default author for rules missing one
+  -snv, --strip-none-values       Strip None values from the rule
   -h, --help                      Show this message and exit.
 ```
 
@@ -290,13 +296,15 @@ Options:
   -id, --rule-id TEXT
   -o, --outfile PATH              Name of file for exported rules
   -r, --replace-id                Replace rule IDs with new IDs before export
-  --stack-version [7.10|7.11|7.12|7.13|7.14|7.15|7.16|7.8|7.9|8.0|8.1|8.2|8.3|8.4|8.5|8.6|8.7|8.8|8.9|8.10|8.11|8.12|8.13|8.14|8.15]
+  --stack-version [7.8|7.9|7.10|7.11|7.12|7.13|7.14|7.15|7.16|8.0|8.1|8.2|8.3|8.4|8.5|8.6|8.7|8.8|8.9|8.10|8.11|8.12|8.13|8.14]
                                   Downgrade a rule version to be compatible
                                   with older instances of Kibana
   -s, --skip-unsupported          If `--stack-version` is passed, skip rule
                                   types which are unsupported (an error will
                                   be raised otherwise)
   --include-metadata              Add metadata to the exported rules
+  -ac, --include-action-connectors
+                                  Include Action Connectors in export
   -e, --include-exceptions        Include Exceptions Lists in export
   -h, --help                      Show this message and exit.
 ```
@@ -332,6 +340,7 @@ Options:
   --kibana-url TEXT
   -kp, --kibana-password TEXT
   -kc, --kibana-cookie TEXT    Cookie from an authed session
+  --api-key TEXT
   --cloud-id TEXT              ID of the cloud instance.
 
 Usage: detection_rules kibana import-rules [OPTIONS]
@@ -344,7 +353,7 @@ Options:
   -id, --rule-id TEXT
   -o, --overwrite                 Overwrite existing rules
   -e, --overwrite-exceptions      Overwrite exceptions in existing rules
-  -a, --overwrite-action-connectors
+  -ac, --overwrite-action-connectors
                                   Overwrite action connectors in existing
                                   rules
   -h, --help                      Show this message and exit.
@@ -512,6 +521,7 @@ Options:
   --kibana-url TEXT
   -kp, --kibana-password TEXT
   -kc, --kibana-cookie TEXT    Cookie from an authed session
+  --api-key TEXT
   --cloud-id TEXT              ID of the cloud instance.
 
 Usage: detection_rules kibana export-rules [OPTIONS]
@@ -520,15 +530,18 @@ Usage: detection_rules kibana export-rules [OPTIONS]
 
 Options:
   -d, --directory PATH            Directory to export rules to  [required]
+  -acd, --action-connectors-directory PATH
+                                  Directory to export action connectors to
   -ed, --exceptions-directory PATH
                                   Directory to export exceptions to
   -r, --rule-id TEXT              Optional Rule IDs to restrict export to
+  -ac, --export-action-connectors
+                                  Include action connectors in export
   -e, --export-exceptions         Include exceptions in export
   -s, --skip-errors               Skip errors when exporting rules
   -sv, --strip-version            Strip the version fields from all rules
   -h, --help                      Show this message and exit.
 
-
 ```
 
 Example of a rule exporting, with errors skipped

diff --git a/detection_rules/action_connector.py b/detection_rules/action_connector.py
@@ -0,0 +1,129 @@
+# Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
+# or more contributor license agreements. Licensed under the Elastic License
+# 2.0; you may not use this file except in compliance with the Elastic License
+# 2.0.
+
+"""Dataclasses for Action."""
+from dataclasses import dataclass
+from datetime import datetime
+from pathlib import Path
+from typing import List, Optional
+
+import pytoml
+from marshmallow import EXCLUDE
+
+from .mixins import MarshmallowDataclassMixin
+from .schemas import definitions
+from .config import parse_rules_config
+
+RULES_CONFIG = parse_rules_config()
+
+
+@dataclass(frozen=True)
+class ActionConnectorMeta(MarshmallowDataclassMixin):
+    """Data stored in an Action Connector's [metadata] section of TOML."""
+
+    creation_date: definitions.Date
+    action_connector_name: str
+    rule_ids: List[definitions.UUIDString]
+    rule_names: List[str]
+    updated_date: definitions.Date
+
+    # Optional fields
+    deprecation_date: Optional[definitions.Date]
+    comments: Optional[str]
+    maturity: Optional[definitions.Maturity]
+
+
+@dataclass
+class ActionConnector(MarshmallowDataclassMixin):
+    """Data object for rule Action Connector."""
+
+    id: str
+    attributes: dict
+    frequency: Optional[dict]
+    managed: Optional[bool]
+    type: Optional[str]
+    references: Optional[List]
+
+
+@dataclass(frozen=True)
+class TOMLActionConnectorContents(MarshmallowDataclassMixin):
+    """Object for action connector from TOML file."""
+
+    metadata: ActionConnectorMeta
+    action_connectors: List[ActionConnector]
+
+    @classmethod
+    def from_action_connector_dict(
+        cls,
+        actions_dict: dict,
+        rule_list: dict,
+    ) -> "TOMLActionConnectorContents":
+        """Create a TOMLActionContents from a kibana rule resource."""
+        rule_ids = []
+        rule_names = []
+
+        for rule in rule_list:
+            rule_ids.append(rule["id"])
+            rule_names.append(rule["name"])
+
+        # Format date to match schema
+        creation_date = datetime.strptime(actions_dict["created_at"], "%Y-%m-%dT%H:%M:%S.%fZ").strftime("%Y/%m/%d")
+        updated_date = datetime.strptime(actions_dict["updated_at"], "%Y-%m-%dT%H:%M:%S.%fZ").strftime("%Y/%m/%d")
+        metadata = {
+            "creation_date": creation_date,
+            "rule_ids": rule_ids,
+            "rule_names": rule_names,
+            "updated_date": updated_date,
+            "action_connector_name": f"Action Connector {actions_dict.get('id')}",
+        }
+
+        return cls.from_dict({"metadata": metadata, "action_connectors": [actions_dict]}, unknown=EXCLUDE)
+
+    def to_api_format(self) -> List[dict]:
+        """Convert the TOML Action Connector to the API format."""
+        converted = []
+
+        for action in self.action_connectors:
+            converted.append(action.to_dict())
+        return converted
+
+
+@dataclass(frozen=True)
+class TOMLActionConnector:
+    """Object for action connector from TOML file."""
+
+    contents: TOMLActionConnectorContents
+    path: Path
+
+    @property
+    def name(self):
+        return self.contents.metadata.action_connector_name
+
+    def save_toml(self):
+        """Save the action to a TOML file."""
+        assert self.path is not None, f"Can't save action for {self.name} without a path"
+        # Check if self.path has a .toml extension
+        path = self.path
+        if path.suffix != ".toml":
+            # If it doesn't, add one
+            path = path.with_suffix(".toml")
+        with path.open("w") as f:
+            contents_dict = self.contents.to_dict()
+            # Sort the dictionary so that 'metadata' is at the top
+            sorted_dict = dict(sorted(contents_dict.items(), key=lambda item: item[0] != "metadata"))
+            pytoml.dump(sorted_dict, f)
+
+
+def parse_action_connector_results_from_api(results: List[dict]) -> tuple[List[dict], List[dict]]:
+    """Filter Kibana export rule results for action connector dictionaries."""
+    action_results = []
+    non_action_results = []
+    for result in results:
+        if result.get("type") != "action":
+            non_action_results.append(result)
+        else:
+            action_results.append(result)
+
+    return action_results, non_action_results
diff --git a/detection_rules/config.py b/detection_rules/config.py
@@ -186,6 +186,7 @@ class RulesConfig:
     version_lock: Dict[str, dict]
 
     action_dir: Optional[Path] = None
+    action_connector_dir: Optional[Path] = None
     auto_gen_schema_file: Optional[Path] = None
     bbr_rules_dirs: Optional[List[Path]] = field(default_factory=list)
     bypass_version_lock: bool = False
@@ -273,6 +274,8 @@ def parse_rules_config(path: Optional[Path] = None) -> RulesConfig:
             contents['exception_dir'] = base_dir.joinpath(directories.get('exception_dir')).resolve()
         if directories.get('action_dir'):
             contents['action_dir'] = base_dir.joinpath(directories.get('action_dir')).resolve()
+        if directories.get('action_connector_dir'):
+            contents['action_connector_dir'] = base_dir.joinpath(directories.get('action_connector_dir')).resolve()
 
     # version strategy
     contents['bypass_version_lock'] = loaded.get('bypass_version_lock', False)

diff --git a/detection_rules/custom_rules.py b/detection_rules/custom_rules.py
@@ -29,6 +29,11 @@ def create_config_content() -> str:
     config_content = {
         'rule_dirs': ['rules'],
         'bbr_rules_dirs': ['rules_building_block'],
+        'directories': {
+            'action_dir': 'actions',
+            'action_connector_dir': 'action_connectors',
+            'exception_dir': 'exceptions',
+        },
         'files': {
             'deprecated_rules': 'etc/deprecated_rules.json',
             'packages': 'etc/packages.yaml',
@@ -98,6 +103,7 @@ def setup_config(directory: Path, kibana_version: str, overwrite: bool, enable_p
     ]
     directories = [
         directory / 'actions',
+        directory / 'action_connectors',
         directory / 'exceptions',
         directory / 'rules',
         directory / 'rules_building_block',

diff --git a/detection_rules/etc/_config.yaml b/detection_rules/etc/_config.yaml
@@ -19,6 +19,7 @@ normalize_kql_keywords: False
 # directories:
   # action_dir: actions
   # exception_dir: exceptions
+  # action_connector_dir: action_connectors
 
 # to set up a custom rules directory, copy this file to the root of the custom rules directory, which is set
 #   using the environment variable CUSTOM_RULES_DIR

diff --git a/detection_rules/generic_loader.py b/detection_rules/generic_loader.py
@@ -11,6 +11,7 @@
 import pytoml
 
 from .action import TOMLAction, TOMLActionContents
+from .action_connector import TOMLActionConnector, TOMLActionConnectorContents
 from .config import parse_rules_config
 from .exception import TOMLException, TOMLExceptionContents
 from .rule_loader import dict_filter
@@ -19,8 +20,8 @@
 
 RULES_CONFIG = parse_rules_config()
 
-GenericCollectionTypes = Union[TOMLAction, TOMLException]
-GenericCollectionContentTypes = Union[TOMLActionContents, TOMLExceptionContents]
+GenericCollectionTypes = Union[TOMLAction, TOMLActionConnector, TOMLException]
+GenericCollectionContentTypes = Union[TOMLActionContents, TOMLActionConnectorContents, TOMLExceptionContents]
 
 
 def metadata_filter(**metadata) -> Callable[[GenericCollectionTypes], bool]:
@@ -101,9 +102,9 @@ def _assert_new(self, item: GenericCollectionTypes) -> None:
         file_map = self.file_map
         name_map = self.name_map
 
-        assert not self.frozen, f"Unable to add item {item.name} {item.id} to a frozen collection"
+        assert not self.frozen, f"Unable to add item {item.name} to a frozen collection"
         assert item.name not in name_map, \
-            f"Rule Name {item.name} for {item.id} collides with rule ID {name_map.get(item.name).id}"
+            f"Rule Name {item.name} collides with {name_map[item.name].name}"
 
         if item.path is not None:
             item_path = item.path.resolve()
@@ -118,9 +119,18 @@ def add_item(self, item: GenericCollectionTypes) -> None:
 
     def load_dict(self, obj: dict, path: Optional[Path] = None) -> GenericCollectionTypes:
         """Load a dictionary into the collection."""
-        is_exception = True if 'exceptions' in obj else False
-        contents = TOMLExceptionContents.from_dict(obj) if is_exception else TOMLActionContents.from_dict(obj)
-        item = TOMLException(path=path, contents=contents)
+        if 'exceptions' in obj:
+            contents = TOMLExceptionContents.from_dict(obj)
+            item = TOMLException(path=path, contents=contents)
+        elif 'actions' in obj:
+            contents = TOMLActionContents.from_dict(obj)
+            item = TOMLAction(path=path, contents=contents)
+        elif 'action_connectors' in obj:
+            contents = TOMLActionConnectorContents.from_dict(obj)
+            item = TOMLActionConnector(path=path, contents=contents)
+        else:
+            raise ValueError("Invalid object type")
+
         self.add_item(item)
         return item
 
@@ -178,6 +188,8 @@ def default(cls) -> 'GenericCollection':
                 collection.load_directory(RULES_CONFIG.exception_dir)
             if RULES_CONFIG.action_dir:
                 collection.load_directory(RULES_CONFIG.action_dir)
+            if RULES_CONFIG.action_connector_dir:
+                collection.load_directory(RULES_CONFIG.action_connector_dir)
             collection.freeze()
             cls.__default = collection