Partner Teams Production Deployment Guide

This Production Deployment Guide for VRO Partner Teams is intended to establish standards and provide structured guidance around the deployment process for applications hosted within VRO production infrastructure. The scope of this guide outlines the responsibilities and expectations for both the VRO Core team and VRO Partner Teams.

1. VRO Core Team Roles and Responsibilities:

• Maintain and manage the production environments.
• Author and update this deployment guide.
• Ensure the reliability, security, and performance of the production infrastructure.
• Assist partner teams with deployment requests and provide necessary support.

2. Partner Team Roles and Responsibilities:

• Develop the benefits management solutions which VRO hosts, preparing these codebases for deployment.
• Submit Deployment Requests once benefits management solutions are ready for deployment.
• Collaborate with the VRO Core team to ensure smooth and successful deployments.

3. Deployment Prerequisites for Production

Following our unanimous support for trunk-based development (May 29th 2024 - VRO Partner Team Sync), all desired pull requests have been merged into the develop branch, with all deployments occurring from the develop branch (unless specifically stated otherwise).

• Pull Requests have addressed any feedback calling for changes, and all automated tests are passing, including the following which are all enforced by GitHub checks:
  • Code Review: Ensure all code changes have been reviewed and approved
  • Unit Tests: All unit tests must pass.
  • Integration Tests: Ensure all integration tests are passing.
• Perform acceptance testing and end-to-end testing on lower environments unless doing so isn’t practical, in which case, include the justification for skipping this requirement in the deployment request details.

4. Deployment Request Submission and Prioritization

Request Submission:

Partner teams should submit a deployment request through the VRO Deployment Request Form on GitHub. (GitHub is the current mechanism, and we’d like to improve it)

Include relevant details including:

• Brief summary of changes (see Deployment Description section for details)
• Deployment due dates and explanation for reasoning (see the deployment due dates addendum for more)
• Dependencies

Prioritization:

• Requests are prioritized based on urgency, business impact, and readiness.
• Emergency fixes get top priority, followed by scheduled releases, and finally minor updates.

5. Deployment Description

Each deployment request should begin with a concise description of what the deployment will accomplish. This description serves as a common reference point for communication between the deployment engineer and the partner team.

• Summarize deployment purpose clearly, focus on key issues, use relevant keywords, indicate issue type, specify affected components, avoid special characters, and review for accuracy.
• Include any technical details engineering team members might anticipate. (e.g., if there are known failure conditions, or sequencing issues such as event dates to deploy by or to deploy after, or data requirements, etc).

This recommendation aims to allow for direct communication between technical team members. With so much focus on brevity and relevance, the technical concerns section looks to capture anything that technical team members deem important.

6. Testing Expectations for Partner Teams

For teams that are consuming VRO microservices, we document the following testing expectations. For teams that are not consuming microservices, please regard these as recommendations. The VRO core team is ready to work with teams who do not already meet these testing expectations.

• Satisfy LHDI and VA PullRequest Testing policies, which includes:
  • Unit Testing: Ensure that new and changed code paths are adequately tested, demonstrating clear and comprehensive test coverage.
  • Integration Testing: Comprehensive integration testing with dependent services.
  • End-to-End Testing: Validate the entire workflow in a lower environment.
  • Collaboration with the VRO team to define a series of post-deploy validation tasks.
• Support VRO definition of Kubernetes Liveness and Readiness Checks: VRO will be responsible for these checks that validate the liveness and readiness of each deployed pod. Checks are defined in helm script and committed to IaC.
• Security Testing: SecRel already enforces that our projects meet the security assessment requirements, and to reiterate what that means:
  • Security assessments are conducted regularly
  • Comprehensive security scans with no unacknowledged vulnerabilities.
  • Consider how we might include penetration testing in the future

7. Post-Deployment Follow-Up

• Communication: Immediately inform the partner team when deployment is complete. Each partner team has a Slack channel and communications regarding deployment status should occur in these respective channels:
  • benefits-employee-exp using @employee-exp-team
  • benefits-contention-classification using @cc-team
• The VRO team is committed to making improvements to our external communication, and will continue to update this document when improving our communication
• Verification: Perform our predetermined smoke test verification.
• Monitoring: Monitor the application for any issues, checking Datadog and Lens for any alerts Kubernetes might be emitting.

For more details of the follow-up processes mentioned above, please see the associated addendum.