Setup for 5xx Error Alerts

[pluckyswan]

Description

Added terraform for 5xx alerts for sftp ingestion service. Tested in internal triggering a 500 error to see Slack activation and deactivation alert.

Issue

CDCgov/trusted-intermediary#1394

Checklist

Note: You may remove items that are not applicable

[github-actions]

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

⏱️ Estimated effort to review: 2 🔵🔵⚪⚪⚪

🧪 No relevant tests

🔒 No security concerns identified

⚡ Recommended focus areas for review Configuration Consistency The lifecycle configuration to ignore changes to tags might not be consistent with the organization's policy on managing infrastructure as code. This could lead to discrepancies in the environment configuration that are not tracked through version control.

[github-actions]

Consider adding a severity label to the alert definition to help in categorizing and prioritizing alerts. This can be done by adding a new tag for severity in the tags section. [important]

[github-actions]

PR Code Suggestions ✨

Explore these optional code suggestions:

Category	Suggestion	Score
Enhancement	Add a recovery action to notify when the system status returns to normal Consider adding a recovery action or notification when the number of 5XX errors falls below the threshold to inform stakeholders when the system status returns to normal. operations/template/alert.tf [64-66] action { action_group_id = data.azurerm_monitor_action_group.notify_slack_email[count.index].id + # Consider adding a recovery action here } Suggestion importance[1-10]: 6 Why: Adding a recovery action is a valuable enhancement for better monitoring and notification management. This suggestion could significantly improve the alert's functionality by informing stakeholders of status normalization.	6
Possible issue	Adjust the condition to correctly enable the alert in the intended environments Ensure that the count condition for creating the "azure_5XX_alert" resource is correctly set to enable the alert in non-production environments only. If the intention is to enable this alert in production environments, the condition should be reversed. operations/template/alert.tf [48] -count = local.non_pr_environment ? 1 : 0 +count = local.non_pr_environment ? 0 : 1 Suggestion importance[1-10]: 5 Why: The suggestion to review the 'count' condition is valid as it impacts the deployment of the alert in different environments. However, without specific knowledge of the intended deployment strategy, it's difficult to assess if the suggested change is correct, hence a moderate score.	5
	Confirm the accuracy of the metric namespace and name for the alert Verify the metric namespace and name to ensure they correctly target the desired Azure resource and metric for monitoring HTTP 5XX errors. operations/template/alert.tf [57-58] metric_namespace = "Microsoft.Web/sites" -metric_name = "Http5xx" +metric_name = "Http5xx" # Confirm this is the correct metric name for monitoring Suggestion importance[1-10]: 4 Why: The suggestion to verify the metric namespace and name is a good practice to ensure correct monitoring setup. However, it does not directly improve or correct the code, rather it prompts a verification step.	4
Best practice	Ensure critical tags are not ignored in the lifecycle configuration Review the list of ignored changes in the lifecycle block to ensure that no critical tags that might affect resource management or compliance are being overlooked. operations/template/alert.tf [70-82] ignore_changes = [ tags["business_steward"], ... tags["zone"] + # Ensure no critical tags are ignored ] Suggestion importance[1-10]: 4 Why: Reviewing ignored tags is a best practice to ensure important configurations are not inadvertently skipped. However, this suggestion is more about caution and does not directly address a specific issue in the PR.	4

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud

[halprin]

Looks hot to trot!