Suggestion: Elastic Common Schema support for match_body in elastalert-create-index

[DrewHorrigan]

Hi. Our analyst is trying to write an Elastalert rule that would query rule matches, creating secondary matches based on rule names and some match_body fields, and I believe we're not the first ones to try that. He ended up hitting a dead end due to the match_body field being disabled in EA writeback index - my understanding is that this was done because it may contain data from different indices with incompatible mappings.

This seems to be a perfect use case for Elastic Common Schema, which has a predefined list of field names, with mappings supplied by Elastic, so I suggest adding a key for elastalert-create-index to include ECS mappings of the appropriate version in the writeback index.

I ended up with the following query to add mappings for ECS-compliant fields, while disabling all others:

PUT elastalert
{
  "mappings" : {
      "dynamic_templates" : [
        {
          "disable_non_ecs_match_body_objects" : {
            "path_match" : "^match_body\\..*",
            "path_unmatch" : "^match_body\\.(agent|client|cloud|container|data_stream|destination|device|dll|dns|ecs|email|error|event|faas|file|group|host|http|log|network|observer|orchestrator|organization|package|process|registry|related|rule|server|service|source|threat|tls|tracing|url|user|user_agent|vulnerability)$",
            "match_mapping_type" : "object",
            "match_pattern" : "regex",
            "mapping" : {
              "enabled" : false
            }
          }
        },
        {
          "disable_non_ecs_match_body_fallback" : {
            "path_match" : "^match_body\\..*",
            "path_unmatch" : "^match_body\\.(agent|client|cloud|container|data_stream|destination|device|dll|dns|ecs|email|error|event|faas|file|group|host|http|log|network|observer|orchestrator|organization|package|process|registry|related|rule|server|service|source|threat|tls|tracing|url|user|user_agent|vulnerability)$",
            "match_pattern" : "regex",
            "mapping" : {
              "type": "keyword",
              "index": false
            }
          }
        },
        {
          "strings_as_keyword" : {
            "match_mapping_type" : "string",
            "mapping" : {
              "ignore_above" : 1024,
              "type" : "keyword"
            }
          }
        }
      ],
      "date_detection" : false,
      "numeric_detection" : true,
      "properties" : {
        "@timestamp" : {
          "type" : "date",
          "format" : "date_optional_time"
        },
        "aggregate_id" : {
          "type" : "keyword"
        },
        "alert_info" : {
          "properties" : {
            "recipients" : {
              "type" : "keyword",
              "ignore_above" : 1024
            },
            "type" : {
              "type" : "keyword",
              "ignore_above" : 1024
            }
          }
        },
        "alert_sent" : {
          "type" : "boolean"
        },
        "alert_time" : {
          "type" : "date",
          "format" : "date_optional_time"
        },
        "category" : {
          "type" : "keyword",
          "ignore_above" : 1024
        },
        "description" : {
          "type" : "keyword",
          "ignore_above" : 1024
        },
        "match_time" : {
          "type" : "date",
          "format" : "date_optional_time"
        },
        "owner" : {
          "type" : "keyword",
          "ignore_above" : 1024
        },
        "priority" : {
          "type" : "long"
        },
        "rule_name" : {
          "type" : "keyword"
        },
        "match_body" : {
            "type": "object",
            "properties": {
            # Everything from https://github.com/elastic/ecs/blob/main/generated/elasticsearch/composable/component/*.json > template.mappings.properties
            }
        }
      }
    }
}

In this template, disable_non_ecs_match_body_objects disables the object-type non-ECS subfields, and disable_non_ecs_match_body_fallback (with the same regex) disables indexing for all non-object non-ECS subfields and also maps them to keyword to prevent mapping type collisions (they still happen even if the field indexing is disabled).

Caveats that I'm aware of:

• The repository I linked to does not seem to contain all ECS fields, though what's available seems to be enough for most use cases.
• ECS version matching may be a bit wonky - it seems that it had its own versioning up until v1.12, and then jumped to 8.0, apparently matching the Elastic Stack version from that point on.

If this seems like a worthwhile addition, I can try to implement it.

[jertel]

Hello!

My experience with ECS is that it works reasonably well as long as all ingested data complies. I've seen some fields that happen to share similar names with ECS but actually contain unrelated data and conflicting types. Ex: agent.name could be a useragent (e.g. a browser like chrome) or it could be an agent service running on user workstations (ex: wazuh, fleet, or even the new Elastic Agent).

I like the idea, and if it makes ElastAlert 2 more usable, more powerful, then I'm onboard. That said, I'd like to see the ECS fields disabled (not created during index creation) by default. So a runtime or config parameter would be necessary to enable that functionality. I think this is best since the majority of users do not attempt to use ElastAlert 2 index data in their rule filters, and so this will eliminate any risk of incidental field conflicts, potentially causing ElastAlert 2 to malfunction and cause people to miss important alerts. Meanwhile, for those power users that know there will never be conflicting field data types, they can enable it (and rebuild their indices if retrofitting an existing install.)

Thanks for offering to implement this functionality. If you end up proceeding, please follow the project's contribution guidelines.