Merge branch 'release-0.11' into pipeline-conditions

.github/workflows/release.yml

@@ -8,7 +8,7 @@ on:
 
 jobs:
   build-and-publish:
-    runs-on: ubuntu-20.04
+    runs-on: ubuntu-24.04
     environment: release
     permissions:
       id-token: write
@@ -17,9 +17,9 @@ jobs:
       - name: Install Poetry
         run: pipx install poetry
       - name: Set up Python
-        uses: actions/setup-python@v4
+        uses: actions/setup-python@v5
         with:
-          python-version: 3.8
+          python-version: 3.9
           cache: poetry
       - name: Verify versioning
         run: |

.github/workflows/test.yml

@@ -10,8 +10,8 @@ jobs:
   test:
     strategy:
       matrix:
-        os: ["ubuntu-20.04", "windows-2019", "macos-12"]
-        python-version: ["3.8", "3.9", "3.10", "3.11"]
+        os: ["ubuntu-24.04", "windows-2022", "macos-14"]
+        python-version: ["3.9", "3.10", "3.11", "3.12", "3.13"]
     runs-on: ${{ matrix.os }}
     steps:
       - uses: actions/checkout@v4

docs/Processing_Pipelines.rst

@@ -99,18 +99,25 @@ Transformation items are defined as a map as follows:
   be used in future conditions.
 * `type`: the type of the transformation as specified in the identifier to class mappings below: :ref:`transformations`
 * Arbitrary transformation parameters are specified at the samle level.
-* `rule_conditions`, `detection_item_conditions`, `field_name_conditions`: conditions of the type corresponding to the name.
+* `rule_conditions`, `detection_item_conditions`, `field_name_conditions`: conditions of the type
+  corresponding to the name. This can be a list of unnamed conditions that are logically linked with
+  the same operator specified in `*_cond_op` or named conditions that are referenced in the
+  `*_cond_expr` attribute. 
 
 Conditions are specified as follows:
 
 * `type`: defines the condition type. It must be one of the identifiers that are defined in
   :ref:`conditions`
 * `rule_cond_op`, `detection_item_cond_op`, `field_name_cond_op`: boolean operator for the condition
-  result. Must be one of `or` or `and`. Defaults to `and`.
+  result. Must be one of `or` or `and`. Defaults to `and`. Alternatively,
+* `rule_cond_expr`, `detection_item_cond_expr`, `field_name_cond_expr`: specify a boolean expression
+  that references to named condition items.
 * `rule_cond_not`, `detection_item_cond_not`, `field_name_cond_not`: if set to *True*, the condition
   result is negated.
 * Arbitrary conditions parameters are specified on the same level.
 
+Specification of an operator and expression is mutually exclusive. 
+
 Example:
 
 .. code-block:: yaml
@@ -149,12 +156,17 @@ There are three types of conditions:
   `ProcessingPipeline`. These can only be applied in the rule pre-processing stage and are evaluated
   only for transformations that operate on field names. 
 
-In addition to the `*_conditions` attributes of `ProcessingPipeline` objects, there are two further
+Conditions can be specified unnamed as list that are logically linked with the operator specified in
+`*_condition_linking` attributes or named as dict that are referenced in the `*_condition_expression`.
+
+In addition to the `*_conditions` attributes of `ProcessingPipeline` objects, there are further
 attributes that control the condition matching behavior:
 
 * `rule_condition_linking`, `detection_item_condition_linking` and `field_name_condition_linking`:
   one of `any` or `all` functions. Controls if one or all of the conditions from the list must match
   to result in an overall match.
+* `rule_condition_expression`, `detection_item_condition_expression` and
+  `field_name_condition_expression`: a boolean expression that references to named condition items.
 * `rule_condition_negation`, `detection_item_condition_negation` and
   `field_name_condition_negation`: if set to *True*, the condition result is negated.
 
@@ -253,9 +265,10 @@ definitions are available:
    "Identifier", "Class"
    "field_name_mapping", "FieldMappingTransformation"
    "field_name_prefix_mapping", "FieldPrefixMappingTransformation"
+   "field_name_transform", "FieldFunctionTransformation"
+   "drop_detection_item", "DropDetectionItemTransformation"
    "field_name_suffix", "AddFieldnameSuffixTransformation"
    "field_name_prefix", "AddFieldnamePrefixTransformation"
-   "drop_detection_item", "DropDetectionItemTransformation"
    "wildcard_placeholders", "WildcardPlaceholderTransformation"
    "value_placeholders", "ValueListPlaceholderTransformation"
    "query_expression_placeholders", "QueryExpressionPlaceholderTransformation"
@@ -266,12 +279,14 @@ definitions are available:
    "set_field", "SetFieldTransformation"
    "replace_string", "ReplaceStringTransformation"
    "map_string", "MapStringTransformation"
+   "set_state", "SetStateTransformation"
    "regex", "RegexTransformation"
    "set_value", "SetValueTransformation"
-   "convert_type", "ConvertTypeTransformation
-   "set_state", "SetStateTransformation"
+   "convert_type", "ConvertTypeTransformation"
    "rule_failure", "RuleFailureTransformation"
    "detection_item_failure", "DetectionItemFailureTransformation"
+   "set_custom_attribute", "SetCustomAttributeTransformation"
+   "nest", "NestedProcessingTransformation"
 
 .. autoclass:: sigma.processing.transformations.FieldMappingTransformation
 
@@ -292,9 +307,10 @@ and `cmdline`. For the latter, OR-conditions will be generated to match the valu
 This is useful if different data models are used in the same system.
 
 .. autoclass:: sigma.processing.transformations.FieldPrefixMappingTransformation
+.. autoclass:: sigma.processing.transformations.FieldFunctionTransformation
+.. autoclass:: sigma.processing.transformations.DropDetectionItemTransformation
 .. autoclass:: sigma.processing.transformations.AddFieldnameSuffixTransformation
 .. autoclass:: sigma.processing.transformations.AddFieldnamePrefixTransformation
-.. autoclass:: sigma.processing.transformations.DropDetectionItemTransformation
 .. autoclass:: sigma.processing.transformations.WildcardPlaceholderTransformation
 .. autoclass:: sigma.processing.transformations.ValueListPlaceholderTransformation
 .. autoclass:: sigma.processing.transformations.QueryExpressionPlaceholderTransformation
@@ -318,6 +334,31 @@ YAML example:
         - mapped2A
         - mapped2B
 
+.. autoclass:: sigma.processing.transformations.SetStateTransformation
+.. autoclass:: sigma.processing.transformations.RegexTransformation
+.. autoclass:: sigma.processing.transformations.SetValueTransformation
+.. autoclass:: sigma.processing.transformations.ConvertTypeTransformation
+.. autoclass:: sigma.processing.transformations.RuleFailureTransformation
+.. autoclass:: sigma.processing.transformations.DetectionItemFailureTransformation
+.. autoclass:: sigma.processing.transformations.SetCustomAttributeTransformation
+.. autoclass:: sigma.processing.transformations.NestedProcessingTransformation
+
+YAML example:
+
+.. code-block:: yaml
+
+  transformations:
+    type: nest
+    items:
+      - type: field_name_mapping
+        mapping:
+          EventID: EventCode
+          CommandLine:
+            - command_line
+            - cmdline
+      - type: set_state
+        state: processed
+
 .. autoclass:: sigma.processing.transformations.RegexTransformation
 .. autoclass:: sigma.processing.transformations.SetValueTransformation
 .. autoclass:: sigma.processing.transformations.ConvertTypeTransformation
@@ -339,12 +380,14 @@ Query Post-Processing Transformations
    "template", "QueryTemplateTransformation"
    "json", "EmbedQueryInJSONTransformation"
    "replace", "ReplaceQueryTransformation"
+   "nest", "NestedQueryPostprocessingTransformation"
 
 .. autoclass:: sigma.processing.postprocessing.EmbedQueryTransformation
 .. autoclass:: sigma.processing.postprocessing.QuerySimpleTemplateTransformation
 .. autoclass:: sigma.processing.postprocessing.QueryTemplateTransformation
 .. autoclass:: sigma.processing.postprocessing.EmbedQueryInJSONTransformation
 .. autoclass:: sigma.processing.postprocessing.ReplaceQueryTransformation
+.. autoclass:: sigma.processing.postprocessing.NestedQueryPostprocessingTransformation
 
 Output Finalization Transformations
 ====================================
@@ -359,11 +402,13 @@ Output Finalization Transformations
    "template", "TemplateFinalizer"
    "json", "JSONFinalizer"
    "yaml", "YAMLFinalizer"
+   "nested", "NestedFinalizer"
 
 .. autoclass:: sigma.processing.finalization.ConcatenateQueriesFinalizer
 .. autoclass:: sigma.processing.finalization.TemplateFinalizer
 .. autoclass:: sigma.processing.finalization.JSONFinalizer
 .. autoclass:: sigma.processing.finalization.YAMLFinalizer
+.. autoclass:: sigma.processing.finalization.NestedFinalizer
 
 Base Classes
 ============

docs/Rule_Validation.rst

@@ -107,6 +107,19 @@ applied to the rule. Example:
 This exclusion defines that the *wildcards_instead_of_modifiers* validator check is disabled for the
 rule with the identifier *5013332f-8a70-4e04-bcc1-06a98a2cca2e*.
 
+Configuration
+-------------
+
+Validator checks that accept parameters can be configured with a dictionary that is passed as the
+*config* parameter. This dictionary maps validator identifiers to dictionaries of parameter-value
+pairs that are passed as keyword arguments to the validator constructor. Example:
+
+.. code-block:: yaml
+
+    config:
+        description_length:
+            min_length: 100
+
 Validator Checks
 ****************
 
@@ -148,6 +161,17 @@ desired rule part and takes care of the proper iteration of these parts. These c
 * :py:class:`sigma.validators.base.SigmaTagValueValidator` for checking all tags appearing beloe the
   *tags* attribute of a Sigma rule.
 
+Parametrization of Checks
+=========================
+
+If required, checks can be parametrized by passing parameters as keyword arguments to the validator
+check constructor. for this purpose, the validator check class must be a *frozen dataclass*. This
+can be achieved by decorating the class with `@dataclass(frozen=True)` from the *dataclasses*
+module.
+
+The parameters can then be specified as dataclass members. The `SigmaValidator` instance will pass
+the parameters to the validator check constructor as keyword arguments.
+
 Base Classes
 ============
 

docs/index.rst

@@ -14,7 +14,8 @@ pySigma Documentation
 Overview
 ********
 
-pySigma is a Python library for parsing, transformation and conversion of Sigma rules. The following
+pySigma is a Python library for parsing, transformation and conversion of Sigma rules. All
+functionality is located within the `sigma` package. The following
 picture summarizes the conversion process from a Sigma rule into a query, which is covered by pySigma:
 
 .. image:: images/conversion.png