[FR] Generate investigation guides

detection_rules/rule_formatter.py

@@ -249,6 +249,14 @@ def _do_write(_data, _contents):
                 # This will ensure that the output file has the correct number of backslashes.
                 v = v.replace("\\", "\\\\")
 
+            if k == 'osquery' and isinstance(v, list):
+                # Specifically handle transform.osquery queries
+                for osquery_item in v:
+                    if 'query' in osquery_item and isinstance(osquery_item['query'], str):
+                        # Transform instances of \ to \\ as calling write will convert \\ to \.
+                        # This will ensure that the output file has the correct number of backslashes.
+                        osquery_item['query'] = osquery_item['query'].replace("\\", "\\\\")
+
             if isinstance(v, dict):
                 bottom[k] = OrderedDict(sorted(v.items()))
             elif isinstance(v, list):

rules/apm/apm_403_response_to_a_post.toml

@@ -2,7 +2,7 @@
 creation_date = "2020/02/18"
 integration = ["apm"]
 maturity = "production"
-updated_date = "2024/05/21"
+updated_date = "2025/01/08"
 
 [rule]
 author = ["Elastic"]
@@ -32,4 +32,51 @@ type = "query"
 query = '''
 http.response.status_code:403 and http.request.method:post
 '''
+note = """## Triage and analysis
+
+### Disclaimer
+
+This investigation guide was generated using generative AI technology and has been reviewed to improve its accuracy and relevance. While every effort has been made to ensure its quality, we recommend validating the content and adapting it to suit your specific environment and operational needs.
+
+### Investigating Web Application Suspicious Activity: POST Request Declined
+
+Web applications often use POST requests to handle data submissions securely. However, adversaries may exploit this by attempting unauthorized actions, triggering a 403 error when access is denied. The detection rule identifies such anomalies by flagging POST requests that receive a 403 response, indicating potential misuse or probing by attackers.
+
+### Possible investigation steps
+
+- Review the specific POST request details, including the URL and parameters, to understand what action was attempted and why it might have been denied.
+- Examine the source IP address of the POST request to determine if it is associated with known malicious activity or if it is an unusual source for your application.
+- Check the user-agent string in the request headers to identify if the request was made by a legitimate browser or a suspicious tool or script.
+- Analyze the frequency and pattern of 403 responses from the same IP or user-agent to identify potential probing or brute force attempts.
+- Investigate the timing of the POST request in relation to other suspicious activities in the logs to identify any correlation with other alerts or incidents.
+- Use Osquery to gather additional context from the server hosting the web application. For example, run the following query to check for recent changes in web server configuration files that might affect POST request handling:
+  ```sql
+  SELECT * FROM file WHERE path LIKE '/etc/httpd/conf/httpd.conf' OR path LIKE '/etc/nginx/nginx.conf' ORDER BY mtime DESC LIMIT 5;
+  ```
+- Review application logs for any error messages or warnings around the time of the POST request to identify potential misconfigurations or application errors.
+- Check for any recent updates or changes to the web application code that might have inadvertently introduced stricter access controls or bugs.
+- Investigate whether the POST request was part of a legitimate user's session by correlating it with authentication logs and session data.
+- Consult threat intelligence sources to determine if there are any known vulnerabilities or attack patterns associated with the specific endpoint or application version targeted by the POST request.
+
+### False positive analysis
+
+- Legitimate users may trigger a 403 response when they attempt to access restricted areas of a web application without proper permissions, such as administrative sections or user-specific content.
+- Automated scripts or bots used for web scraping or data collection might inadvertently cause 403 errors if they attempt to access protected resources, leading to false positives.
+- Security tools or plugins that perform vulnerability scanning on web applications can generate POST requests that result in 403 responses, which are not necessarily malicious.
+- Users can manage these false positives by creating exceptions for known IP addresses or user agents that frequently cause 403 errors but are verified as non-threatening.
+- Implementing a whitelist for specific endpoints or user roles that are known to trigger 403 responses during normal operations can help reduce unnecessary alerts.
+- Regularly reviewing and updating the list of exceptions based on the analysis of web application logs and user behavior patterns can help maintain the accuracy of the detection rule.
+
+### Response and remediation
+
+- Immediately review the logs to identify the source IP address of the suspicious POST requests and block it at the firewall or web application firewall (WAF) to prevent further unauthorized attempts.
+- Analyze the request payloads to determine if there are any patterns or specific data being targeted, which could indicate the attacker's intent or the vulnerability being exploited.
+- Conduct a thorough review of the web application's access control policies to ensure that permissions are correctly configured and that sensitive endpoints are adequately protected.
+- Escalate the incident to the security operations center (SOC) or incident response team if the activity appears to be part of a larger attack campaign or if multiple systems are affected.
+- Implement enhanced logging for POST requests, including capturing request headers and payloads, to improve visibility and aid in future investigations.
+- Integrate threat intelligence feeds to correlate the detected activity with known attack patterns or threat actors, providing additional context for the investigation.
+- Restore any affected systems or services to their operational state by rolling back unauthorized changes and applying necessary patches or updates to address any identified vulnerabilities.
+- Conduct a post-incident review to identify gaps in the current security posture and update incident response plans accordingly.
+- Implement security hardening measures such as input validation, rate limiting, and multi-factor authentication to reduce the risk of similar incidents in the future.
+- Educate development and operations teams on secure coding practices and the importance of regular security assessments to prevent exploitation of web application vulnerabilities."""
 

rules/apm/apm_405_response_method_not_allowed.toml

@@ -2,7 +2,7 @@
 creation_date = "2020/02/18"
 integration = ["apm"]
 maturity = "production"
-updated_date = "2024/05/21"
+updated_date = "2025/01/08"
 
 [rule]
 author = ["Elastic"]
@@ -32,4 +32,47 @@ type = "query"
 query = '''
 http.response.status_code:405
 '''
+note = """## Triage and analysis
+
+### Disclaimer
+
+This investigation guide was generated using generative AI technology and has been reviewed to improve its accuracy and relevance. While every effort has been made to ensure its quality, we recommend validating the content and adapting it to suit your specific environment and operational needs.
+
+### Investigating Web Application Suspicious Activity: Unauthorized Method
+
+Web applications use HTTP methods to define actions on resources. A 405 status code signals a method is not allowed, often due to misconfigurations or probing by attackers. Adversaries may exploit this by testing various methods to find vulnerabilities. The detection rule identifies such attempts by flagging responses with a 405 status, indicating potential unauthorized method usage.
+
+### Possible investigation steps
+
+- Review the logs for the specific time frame when the 405 status code was triggered to identify any patterns or anomalies in the requests.
+- Examine the source IP addresses associated with the 405 responses to determine if they are known or potentially malicious.
+- Analyze the user-agent strings in the requests to identify if they are associated with known malicious tools or scripts.
+- Check the HTTP methods used in the requests that resulted in a 405 response to understand which methods were attempted.
+- Investigate the requested URLs or endpoints to determine if they are legitimate resources or if they appear to be probing for vulnerabilities.
+- Correlate the 405 response events with other security events or alerts to identify any related suspicious activity.
+- Use Osquery to gather additional context from the web server, such as running processes or open network connections. Example query: `SELECT * FROM processes WHERE name LIKE '%httpd%' OR name LIKE '%nginx%';`
+- Review the web application configuration files to ensure that the HTTP methods are correctly configured and that there are no misconfigurations.
+- Check for any recent changes or updates to the web application or server that might have affected the allowed HTTP methods.
+- Consult threat intelligence sources to determine if there are any known campaigns or threat actors associated with similar activity.
+
+### False positive analysis
+
+- Legitimate applications or services may trigger a 405 response when they are misconfigured or when certain HTTP methods are intentionally disabled for security reasons. 
+- Automated tools or scripts used for testing or monitoring web applications might inadvertently use unsupported HTTP methods, resulting in 405 responses.
+- Web crawlers or bots that are not malicious but are poorly configured can also cause 405 status codes by attempting to access resources with incorrect methods.
+- To manage these false positives, users can create exceptions for known and trusted IP addresses or user agents that frequently trigger 405 responses without malicious intent.
+- Regularly review and update the list of exceptions to ensure that only non-threatening behaviors are excluded, maintaining a balance between security and operational efficiency.
+
+### Response and remediation
+
+- Immediately review the web server logs to identify the source IP addresses and user agents associated with the 405 status code responses to determine if they are part of a legitimate request or a potential attack.
+- Block or rate-limit the identified suspicious IP addresses at the firewall or web application firewall (WAF) to prevent further unauthorized method attempts.
+- Conduct a thorough investigation to determine if the unauthorized method attempts were successful in exploiting any vulnerabilities, and assess the potential impact on the system.
+- Review and update the web application's configuration to ensure that only necessary HTTP methods are allowed for each resource, reducing the attack surface.
+- Escalate the incident to the security operations team if there is evidence of a coordinated attack or if sensitive data may have been exposed.
+- Implement enhanced logging policies to capture detailed information about HTTP requests, including headers and payloads, to aid in future investigations.
+- Integrate security information and event management (SIEM) systems with web server logs to enable real-time monitoring and alerting of suspicious activities.
+- Restore the system to its operational state by applying patches or updates to address any identified vulnerabilities and ensure the web application is functioning correctly.
+- Conduct a post-incident review to analyze the attack vectors and improve the incident response plan, incorporating lessons learned from the investigation.
+- Implement hardening measures such as disabling unused HTTP methods, enforcing strong authentication and authorization controls, and regularly testing the web application for vulnerabilities."""
 

rules/apm/apm_sqlmap_user_agent.toml

@@ -2,7 +2,7 @@
 creation_date = "2020/02/18"
 integration = ["apm"]
 maturity = "production"
-updated_date = "2024/05/21"
+updated_date = "2025/01/08"
 
 [rule]
 author = ["Elastic"]
@@ -32,4 +32,50 @@ type = "query"
 query = '''
 user_agent.original:"sqlmap/1.3.11#stable (http://sqlmap.org)"
 '''
+note = """## Triage and analysis
+
+### Disclaimer
+
+This investigation guide was generated using generative AI technology and has been reviewed to improve its accuracy and relevance. While every effort has been made to ensure its quality, we recommend validating the content and adapting it to suit your specific environment and operational needs.
+
+### Investigating Web Application Suspicious Activity: sqlmap User Agent
+
+Sqlmap is a widely-used open-source tool designed to automate the detection and exploitation of SQL injection vulnerabilities in web applications. While beneficial for security testing, adversaries can exploit it to gain unauthorized access to databases. The detection rule identifies suspicious activity by flagging the specific user agent string associated with sqlmap, helping security analysts pinpoint potential misuse.
+
+### Possible investigation steps
+
+- Review the alert details to confirm the presence of the sqlmap user agent string: `sqlmap/1.3.11#stable (http://sqlmap.org)` in the `user_agent.original` field.
+- Correlate the timestamp of the alert with web server logs to identify the source IP address and any associated requests.
+- Investigate the source IP address to determine if it is known for malicious activity or if it belongs to a legitimate user or organization.
+- Examine the request URL and parameters in the web server logs to identify any potential SQL injection attempts or unusual patterns.
+- Check for any other user agents or unusual activity from the same IP address around the time of the alert to identify patterns of behavior.
+- Use Osquery to gather additional context on the system that received the request. For example, run the following Osquery query to list recent web server access logs:
+  ```sql
+  SELECT * FROM apache_access WHERE user_agent = 'sqlmap/1.3.11#stable (http://sqlmap.org)';
+  ```
+- Investigate any database logs or alerts for suspicious queries or access patterns that coincide with the time of the alert.
+- Review any recent changes or updates to the web application that might have introduced vulnerabilities or altered security configurations.
+- Check for any other alerts or incidents involving the same user agent or IP address across different systems or applications.
+- Consult threat intelligence sources to determine if there are any known campaigns or threat actors currently using sqlmap for malicious purposes.
+
+### False positive analysis
+
+- Legitimate security teams or developers may use sqlmap for authorized penetration testing, leading to false positives when the tool is used in a controlled environment.
+- Automated security scans or vulnerability assessments conducted by third-party services might trigger the rule if sqlmap is part of their testing suite.
+- To manage these false positives, organizations can create exceptions for known IP addresses or user agents associated with authorized security testing activities.
+- Implementing a whitelist of IP addresses or user agents for trusted security tools can help reduce noise and focus on genuine threats.
+- Regularly review and update the list of exceptions to ensure that only current and authorized activities are excluded from alerts.
+
+### Response and remediation
+
+- Immediately block the IP address associated with the suspicious sqlmap user agent activity to prevent further unauthorized access attempts.
+- Conduct a thorough investigation of the affected web application logs to identify any successful SQL injection attempts and assess the extent of potential data exposure.
+- Review and update web application firewall (WAF) rules to detect and block SQL injection attempts more effectively.
+- Escalate the incident to the security operations center (SOC) for further analysis and to determine if the activity is part of a larger attack campaign.
+- Implement enhanced logging policies to capture detailed user agent strings and other relevant HTTP request headers for future analysis.
+- Integrate threat intelligence feeds to correlate detected sqlmap activity with known malicious IP addresses and attack patterns.
+- Restore any compromised databases from the most recent clean backup and ensure that all patches and updates are applied to the database management system.
+- Conduct a security review of the web application code to identify and remediate any SQL injection vulnerabilities.
+- Educate development and operations teams on secure coding practices and the importance of input validation to prevent SQL injection attacks.
+- Consider deploying additional security measures such as intrusion detection systems (IDS) and regular vulnerability assessments to strengthen the overall security posture."""