5xx error alerts

[pluckyswan]

5xx error alerts

Added terraform for 5xx error Azure alerts. Because it was deemed as higher priority than 4xx errors, the threshold is lower and more frequent. It was tested in internal by creating and hitting a temporary bad health check that returns a 500. Alert appeared in Slack when activated and deactivated.

Issue

#1394

Checklist

Note: You may remove items that are not applicable

[github-actions]

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

🎫 Ticket compliance analysis 🔶 1394 - Partially compliant Fully compliant requirements: Alerts for 5xx status codes should be implemented. Not compliant requirements: Documentation and diagrams should be created or updated. Code should be refactored for clarity and no design/technical debt. Security & privacy gates should be passed. Unit test coverage of our code should be >= 90%.

⏱️ Estimated effort to review: 3 🔵🔵🔵⚪⚪

🧪 No relevant tests

🔒 No security concerns identified

⚡ Recommended focus areas for review Incomplete Implementation The PR includes placeholders for alert frequency and history window size which are marked as TBD. These need to be finalized.

[github-actions]

PR Code Suggestions ✨

Explore these optional code suggestions:

Category	Suggestion	Score
Enhancement	Finalize the configuration for alert frequency and history window size Remove the placeholder comment "//TBD: How frequent do we want this alert and how far do we want it to look back." and finalize the configuration values for frequency and window_size to ensure the alert system is fully operational upon deployment. operations/template/alert.tf [100-130] -//TBD: How frequent do we want this alert and how far do we want it to look back. +# Configuration finalized for frequency and window_size. Suggestion importance[1-10]: 7 Why: Removing the placeholder comment and finalizing the configuration values are crucial for ensuring the alert system is fully operational upon deployment. This suggestion directly impacts the functionality and readiness of the alert system.	7
	Adjust the threshold for HTTP 4XX and 5XX alerts to reduce noise from expected or minor issues Consider adding more specific criteria for the HTTP 4XX and 5XX alerts to avoid triggering alerts for expected or minor issues, potentially by setting a higher threshold or by specifying additional conditions. operations/template/alert.tf [107-137] -threshold = 1 +threshold = 10 Suggestion importance[1-10]: 6 Why: Adjusting the threshold for HTTP 4XX and 5XX alerts is a valid enhancement to reduce unnecessary noise from minor issues. However, the exact threshold should be determined based on specific operational requirements and data analysis.	6
Possible issue	Ensure action group references are valid and correctly resolved in alert configurations Verify that the action_group_id references in the alert configurations are correctly resolving to valid action groups, especially considering the conditional creation based on count.index. operations/template/alert.tf [37-141] -action_group_id = azurerm_monitor_action_group.notify_slack_email[count.index].id +action_group_id = azurerm_monitor_action_group.notify_slack_email[0].id # Ensure this resolves correctly in all environments Suggestion importance[1-10]: 5 Why: Verifying that the action_group_id resolves correctly is important for ensuring that alerts are properly linked to their action groups. However, the suggestion to change the index from count.index to 0 is not necessarily correct without additional context about the deployment environment and configuration.	5

[halprin]

I see these PRs build on each other. I'd apply the feedback about the tags in the previous PR to this PR too.

[halprin]

Since this PR seems to have been branched off of our 4xx alert branch, I would wait until #1465 gets merged before this is merged.

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud