Stabilize Uptime of Editor Experience

[gracekretschmer-metrostar]

Status

Update each sprint until completed

Date	Status	Launch Date (see above)	Notes
10/29/2024	in-progress	on-track	working on roll-back job now and building up a support rotation

Problem Statement

There have been 4 times (source) over the last 3 months (from July - September 2024) that the editor experience (prod CMS) has been unavailable. Therefore, prod CMS' uptime fell to 99.786% in the last 3 months, which is below the industry standard of 99.9% uptime. To be in alignment with industry standard (99.9% uptime) and maintain trust with editors, we need to put in preventative measures to improve the uptime of the editor experience.

Business Outcomes / OKR

• Objective: Our platforms are the best way to deliver products at VA
• Key Result: Out platforms hit the "elite" level (as defined by DORA) on Deployment Frequency, Lead Time for Changes, Change Failure Rate, and Time to Restore Service

Hypothesis or Bet

• Add preventative measures to mitigate the missing IP addresses
• Add and automate a rollback job into Jenkins that will better automate the needed rollback step when prod CMS goes offline.

We will know we're done when... ("Definition of Done")

• The editor experience uptime increases to 99.9% over a three month period.
• Measures are implemented that decrease the amount of time it takes to bring prod CMS back online when it goes offline.

Known Blockers/Dependencies

List any blockers or dependencies for this work to be completed

Projected Launch Date

End of Q4, 12/31/2024.

Launch Checklist

Guidance (delete before posting)

Is this service / tool / feature...

... tested?

• Usability test (TODO: link) has been performed, to validate that new changes enable users to do what was intended and that these changes don't worsen quality elsewhere. If usability test isn't relevant for this change, document the reason for skipping it.
  • ... and issues discovered in usability testing have been addressed.
  • Note on skipping: metrics that show the impact of before/after can be a substitute for usability testing.
• End-to-end manual QA or UAT is complete, to validate there are no high-severity issues before launching
• (if applicable) New functionality has thorough, automated tests running in CI/CD

... documented?

• New documentation is written pursuant to our documentation style guide
• Product is included in the List of VSP Products
  • List the existing product that this initiative fits within, or add a new product to this list.
• Internal-facing: there's a Product Outline
• External-facing: a User Guide on Platform Website exists for this product/feature tool
• (if applicable) Post to #vsp-service-design for external communication about this change (e.g. customer-facing meetings)

... measurable

• We will be measuring the uptime of prod CMS uptime to meet the goal of 99.9% uptime

When you're ready to launch...

• Configure your communication to teams about your work using the Change Communication Guide
• Conduct a go/no-go when you're almost ready to launch.

Required Artifacts

Documentation

• PRODUCT_NAME: directory name used for your product documentation
• Product Outline: link to Product Outline
• User Guide: link to User Guide

Testing

• Usability test: link to GitHub issue, or provide reason for skipping
• Manual QA: link to GitHub issue or documented results
• Automated tests: link to tests, or "N/A"

Measurement

• https://vagov.ddog-gov.com/dashboard/vnk-g4s-fru/cms-prod--staging?fromUser=false&refr[…]sliding&from_ts=1728570082786&to_ts=1729174882786&live=true - measuring the prod CMS uptime to meet the goal of 99.9% uptime