#376 - Add logging standards from MBTP

docs/patterns/securing-application-logging.md

@@ -19,7 +19,7 @@ related:
 ---
 
 It is common to log user information from an application to assist operations or debugging. Some middleware might be configured to log full configuration, data, or request/response objects, which may contain Personally Identifiable Information (PII), authentication tokens or application secrets.
-
+  
 Logging any of these can make information available to people that have access to log aggregation platforms, or log storage, but are not legally allowed to view PII, or should not otherwise have access to that information. User authentication tokens and application secrets can be used to impersonate users and elevate privilege.
 
 [Sensitive Data Exposure](https://owasp.org/www-project-top-ten/2017/A3_2017-Sensitive_Data_Exposure) has been a recognised risk to secure application development historically. Our [managing secrets]({{ '/standards/managing-secrets/' | url }}) standard has requirements on ensuring logs do not contain tokens or other secrets.
@@ -65,6 +65,18 @@ Logging full payloads should be avoided. However, logging full payloads can be u
 
 ## Considerations
 
+### Consider how information can combine to form PII
+
+Examples of combined PII include:
+* Passport number, Issuing Country, Expiry Dates
+* BRP/C number, Date of Birth
+
+With the examples above, the information by themselves do not mean very much. After all, a BRP/C number is just a "random" sequence of characters; but in combination with the date of birth it can be used to uniquely identify an individual.
+
+If you are unsure what is person identifiable information, consult with your security specialists.
+
+Information and data sets that contain sensitive information should not be output to logs regardless of log level.
+
 ### Consider whether logging statements are necessary
 Sometimes logging is used for fast-feedback loops when prototyping application functionality, but eventually become redundant, or the application they provide does not provide additional operational or debugging value for an operator searching through logs.
 

docs/standards/logging.md

@@ -0,0 +1,73 @@
+---
+layout: standard
+order: 1
+title: Logging for services with centralised logging aggregation
+date: 2024-01-24
+id: SEGAS-00015 
+tags:
+  - SRE
+  - Monitoring
+  - Observability
+  - Logging
+related: 
+  sections:
+    - title: Securing application logging
+      items:
+        - text: Securing application logging
+          href: /patterns/securing-application-logging/ 
+---
+
+Logging tools are one of the primary tools to debug application problems and identifying the root cause of incidents. However, logs need to be formatted in a standard way so that they are not only useful for the application's original developers, but also those supporting the application later in its life-cycle. 
+
+However, logs are sometimes relied on too heavily for problem detection, when this role would be better suited to purpose chosen monitoring tooling. 
+
+Therefore these standards provide guidance on both the proper format of application logs, as well as their proper usage.
+
+---
+
+## Requirements
+
+- [Service logs MUST have the minimum required field](#service-logs-must-have-the-minimum-required-fields)
+- [Service logs MUST be forwarded to log aggregators](#service-logs-must-be-forwarded-to-log-aggregators)
+- [Service logs MUST be rotated daily and no more than 100 MB of logs to be retained](#service-logs-must-be-rotated-daily-and-no-more-than-100-mb-of-logs-to-be-retained)
+- [Service logs MUST have an identifiable source in the log aggregator](#service-logs-must-have-an-identifiable-source-in-the-log-aggregator)
+- [Service log messages should only be used for metrics, dashboarding and alerting as a last resort](#service-log-messages-should-only-be-used-for-metrics%2C-dashboarding-and-alerting-as-a-last-resort)
+- [There MUST be no DEBUG / TRACE level messages in Production environments](#there-must-be-no-debug-%2F-trace-level-messages-in-production-environments)
+- [Service log messages MUST not reveal any sensitive information or person identifiable information](#service-log-messages-must-not-reveal-any-sensitive-information-or-person-identifiable-information)
+
+
+### Service logs MUST have the minimum required fields
+
+Information | Description
+------------| -----------
+ Date / Time | Each log message must include date and time (millisecond accuracy) (UTC). The time source for all service logs must be consistent so that messages can be viewed & sorted easily.
+Service | Each log message must be attributed to an service (i.e. hostname).
+Log level | Each log message must have an appropriate log level (INFO, WARN, ERROR, etc.)
+Log Message | Each log message must have a log message. Log messages must follow a consistent log messaging format for all log types to allow for easy processing and translation of logs to other formats.
+
+### Service logs MUST be forwarded to log aggregators
+
+Service logs must be forwarded to the supported log aggregation tooling for centralised and persistent storage, and ease of querying log information across distributed applications and services.
+
+### Service logs MUST be rotated daily and no more than 100 MB of logs to be retained
+
+Where log files are persisted locally, log files must be rotated to prevent filling up local disk space. Furthermore, no more than 100 MB of logs are to be persisted should your service generate lots of logs.
+
+### Service logs MUST have an identifiable source in the log aggregator
+
+Services can have a variety of logs, including application, error, and access logs; so it must be possible to identify the source of the log data in the log aggregation tooling.
+
+### Service log messages should only be used for metrics, dashboarding and alerting as a last resort
+Although acceptable in some cases, it is strongly recommended that you do not use information contained in log messages as the primary method by which you report on application health, performance and business metrics. Information in logs can be difficult to parse, aggregate and quite brittle.
+
+The use of your approved tooling for service monitoring and alerting should always be prioritised over the use of log-based monitoring.
+
+### There MUST be no DEBUG / TRACE level messages in Production environments
+
+Refer to our [securing application logging]({{ '/patterns/securing-application-logging/#take-care-when-using-detailed-logging-to-support-application-debugging' | url }}) pattern for more details.
+
+### Service log messages MUST not reveal any sensitive information or person identifiable information
+
+Refer to our [securing application logging]({{ '/patterns/securing-application-logging/#consider-how-information-can-combine-to-form-pii' | url }}) pattern for more details.
+
+---