From 8005b1b964c6777d10241ed903aac5ea4bed73a4 Mon Sep 17 00:00:00 2001 From: Matt Jadud Date: Tue, 19 Mar 2024 06:26:29 -0400 Subject: [PATCH 1/6] Suggestions for increased transparency This PR suggests some text changes to the copy, and adds back a chronology. Because our site is used by: 1. Auditors and auditees submitting audits 2. State auditors (and other entities) responsible for passthrough monitoring 3. Federal audit resolution officials 4. IG offices 5. Grants managers/grants issuing offices 6. GAO 8. OMB 9. The general public 10. ... I think credibility is enhanced by transparency, and because we have a public record of our efforts in this space (our Github repository), I think we should use it to document the actions we've taken so far. For consideration. --- src/status.md | 20 +++++++++++++++++--- 1 file changed, 17 insertions(+), 3 deletions(-) diff --git a/src/status.md b/src/status.md index 9aa3b0e3..76f5cd5e 100644 --- a/src/status.md +++ b/src/status.md @@ -11,7 +11,21 @@ meta: **Audit search is currently limited to UEI-only.** You can search your UEI to confirm your audit submission is complete. The PDF report will still be available for download but summary report workbooks won't. ## Issue background -In mid-February, GSA completed the migration of historical data from the Census Bureau. This increased the amount of data in the GSA system exponentially and slowed down search performance. -Our team has released multiple patches to stabilize search performance since then but have been unable to do so consistently. For this reason, we've decided to reduce search to UEI-only while we focus on building new search tables instead of one-off bug fixes. -We're working as quickly and efficiently as possible to return full search functionality to our users as soon as we can. Thank you for your patience. \ No newline at end of file +On February 7th, GSA completed the migration of historical data from the Census Bureau. This increased the amount of data in the GSA system more than twenty-fold, drastically impairing search performance. + +Our team has taken incremental, iterative steps to improve search performance since then but have been unable to do so in a manner that addresses the multiple challenges at hand. For this reason, we've decided to reduce search to UEI-only while we focus on building the infrastructure necessary to support a more sustainable search experience. + +We're working as quickly as possible to return full search functionality to our users as soon as we can. We recognize the significant disruption this represents to users with oversight responsibilities, and will provide daily updates here until this issue is resolved. + +A brief history of our efforts: + +1. February 7th, data from Census is completely loaded, increasing the number of audits from 10K to more than 250K, and the number of awards tracked from 100K to nearly 4.5M. +2. February 8th, [resources increased in production](https://github.com/GSA-TTS/FAC/pull/3376) to address load. +3. February 9th, [My ALN](https://github.com/GSA-TTS/FAC/pull/3378) functionality is removed in an attempt to address crashing behaviors stemming from this feature. +4. February 14th, worked with more than a dozen NSACs to [prioritize search needs](https://github.com/GSA-TTS/FAC/issues/3388). +5. February 16th, [increased search capacity in production](https://github.com/GSA-TTS/FAC/pull/3424). +6. February 28th, [added prioritized search needs](https://github.com/GSA-TTS/FAC/pull/3440), re-introducing performance challenges. +7. March 11th, heavy system load prevents system updates for nearly a week. [Post-workday update to increase resources in production](https://github.com/GSA-TTS/FAC/pull/3496) carried out, partially addressing new search performance challenges. +8. March 14th, performance issues and crashing persist in search. [New search tables and queries introduced](https://github.com/GSA-TTS/FAC/pull/3511). Changes require significant engineering and testing work. +9. March 18th, decision to take down search made by team. \ No newline at end of file From b8ca123785d43cbc057196cafc60c8810fec8a67 Mon Sep 17 00:00:00 2001 From: Matt Jadud Date: Tue, 19 Mar 2024 06:32:48 -0400 Subject: [PATCH 2/6] Simplification in second para. --- src/status.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/status.md b/src/status.md index 76f5cd5e..979e21c7 100644 --- a/src/status.md +++ b/src/status.md @@ -14,9 +14,9 @@ meta: On February 7th, GSA completed the migration of historical data from the Census Bureau. This increased the amount of data in the GSA system more than twenty-fold, drastically impairing search performance. -Our team has taken incremental, iterative steps to improve search performance since then but have been unable to do so in a manner that addresses the multiple challenges at hand. For this reason, we've decided to reduce search to UEI-only while we focus on building the infrastructure necessary to support a more sustainable search experience. +Our team has taken incremental, iterative steps to improve search performance since then but has been unable to quickly address the multiple challenges at hand. For this reason, we've decided to reduce search to UEI-only while we focus on building the infrastructure necessary to support a more sustainable search experience. A UEI-only search allows auditors and auditees to verify that their submissions are complete. -We're working as quickly as possible to return full search functionality to our users as soon as we can. We recognize the significant disruption this represents to users with oversight responsibilities, and will provide daily updates here until this issue is resolved. +We're working as quickly as possible to return full search functionality as soon as we can. We recognize the significant disruption this represents to users with oversight responsibilities, and will provide daily updates here until this issue is resolved. A brief history of our efforts: From d8d2e0ef46bfd005b904cb0b706b01e85f091e0c Mon Sep 17 00:00:00 2001 From: Matt Jadud Date: Tue, 19 Mar 2024 06:36:02 -0400 Subject: [PATCH 3/6] One more. --- src/status.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/status.md b/src/status.md index 979e21c7..48f9a9c0 100644 --- a/src/status.md +++ b/src/status.md @@ -16,7 +16,7 @@ On February 7th, GSA completed the migration of historical data from the Census Our team has taken incremental, iterative steps to improve search performance since then but has been unable to quickly address the multiple challenges at hand. For this reason, we've decided to reduce search to UEI-only while we focus on building the infrastructure necessary to support a more sustainable search experience. A UEI-only search allows auditors and auditees to verify that their submissions are complete. -We're working as quickly as possible to return full search functionality as soon as we can. We recognize the significant disruption this represents to users with oversight responsibilities, and will provide daily updates here until this issue is resolved. +We're working hard to return full search functionality as soon as possible. We recognize the significant disruption this represents to users with oversight responsibilities, and will provide daily updates here until this issue is resolved. A brief history of our efforts: From 6f7079e6fd4975f4c093250e2e69fb00cc71b7bd Mon Sep 17 00:00:00 2001 From: Laura Herring Date: Tue, 19 Mar 2024 08:51:26 -0400 Subject: [PATCH 4/6] copy edits to status page --- src/status.md | 43 +++++++++++++++++++++++++++---------------- 1 file changed, 27 insertions(+), 16 deletions(-) diff --git a/src/status.md b/src/status.md index 48f9a9c0..1fa87387 100644 --- a/src/status.md +++ b/src/status.md @@ -5,27 +5,38 @@ meta: name: FAC system status description: Track the operating status of the FAC, both for audit submission and search. --- +# FAC system status -# March 18, 2024 +This page tracks the operating status of the FAC, both for audit submission and search. -**Audit search is currently limited to UEI-only.** You can search your UEI to confirm your audit submission is complete. The PDF report will still be available for download but summary report workbooks won't. +## March 18, 2024 -## Issue background +**Audit search is currently limited to UEI-only.** You can search your UEI to confirm your audit submission is complete. The PDF report is still be available for download but summary report workbooks aren't. -On February 7th, GSA completed the migration of historical data from the Census Bureau. This increased the amount of data in the GSA system more than twenty-fold, drastically impairing search performance. +### Issue background -Our team has taken incremental, iterative steps to improve search performance since then but has been unable to quickly address the multiple challenges at hand. For this reason, we've decided to reduce search to UEI-only while we focus on building the infrastructure necessary to support a more sustainable search experience. A UEI-only search allows auditors and auditees to verify that their submissions are complete. +In mid-February, GSA completed the migration of historical data from the Census Bureau. This increased the amount of data in the GSA system exponentially, hindering search performance. -We're working hard to return full search functionality as soon as possible. We recognize the significant disruption this represents to users with oversight responsibilities, and will provide daily updates here until this issue is resolved. +Over the last month, our team has released multiple patches to stabilize search with limited success. Because of this, we've reduced search to UEI-only while we focus on building a more sustainable search function. -A brief history of our efforts: +We're working hard to return full search functionality as soon as we can. We know this is a significant disruption for many users. Thank you for your patience. -1. February 7th, data from Census is completely loaded, increasing the number of audits from 10K to more than 250K, and the number of awards tracked from 100K to nearly 4.5M. -2. February 8th, [resources increased in production](https://github.com/GSA-TTS/FAC/pull/3376) to address load. -3. February 9th, [My ALN](https://github.com/GSA-TTS/FAC/pull/3378) functionality is removed in an attempt to address crashing behaviors stemming from this feature. -4. February 14th, worked with more than a dozen NSACs to [prioritize search needs](https://github.com/GSA-TTS/FAC/issues/3388). -5. February 16th, [increased search capacity in production](https://github.com/GSA-TTS/FAC/pull/3424). -6. February 28th, [added prioritized search needs](https://github.com/GSA-TTS/FAC/pull/3440), re-introducing performance challenges. -7. March 11th, heavy system load prevents system updates for nearly a week. [Post-workday update to increase resources in production](https://github.com/GSA-TTS/FAC/pull/3496) carried out, partially addressing new search performance challenges. -8. March 14th, performance issues and crashing persist in search. [New search tables and queries introduced](https://github.com/GSA-TTS/FAC/pull/3511). Changes require significant engineering and testing work. -9. March 18th, decision to take down search made by team. \ No newline at end of file +**A timeline of the issues:** + +- February 7: We completed the data migration. This increased the number of audits from 10K to more than 250K, and the number of awards tracked from 100K to nearly 4.5M. + +- February 8: We [increased resources in our production environment](https://github.com/GSA-TTS/FAC/pull/3376) to address the increased load. + +- February 9: The team removed the [My ALN filter](https://github.com/GSA-TTS/FAC/pull/3378) in an attempt to stop system crashes caused by this feature. + +- February 14: We worked with more than a dozen NSACs to [prioritize search needs](https://github.com/GSA-TTS/FAC/issues/3388). + +- February 16: The team [increased search capacity in the production environment](https://github.com/GSA-TTS/FAC/pull/3424). + +- February 28: We [added prioritized search needs](https://github.com/GSA-TTS/FAC/pull/3440), re-introducing performance challenges. + +- March 11: Heavy use of the system prevented system updates for nearly a week. The team completed [an after-hours update to the production environment](https://github.com/GSA-TTS/FAC/pull/3496), partially addressing new search performance issues. + +- March 14: Performance issues and system crashes persisted in search. The team released [new search tables and queries](https://github.com/GSA-TTS/FAC/pull/3511). These required significant engineering and testing work. + +- March 18: The team decided to take down search to focus on long-term solutions vs. short-term patches. \ No newline at end of file From b6a51535f436dca26de6d2a156362eab1d4d8ae5 Mon Sep 17 00:00:00 2001 From: Laura Herring Date: Tue, 19 Mar 2024 15:09:00 -0400 Subject: [PATCH 5/6] march 19 status update --- src/status.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/src/status.md b/src/status.md index 1fa87387..9d0f44ef 100644 --- a/src/status.md +++ b/src/status.md @@ -9,6 +9,12 @@ meta: This page tracks the operating status of the FAC, both for audit submission and search. +## March 19 + +We're aware that search performance, even reduced to UEI-only, continues to be slow. Our engineers are investigating the root cause of the issue and continue to work on a solution. + +We are also exploring options for using the API to generate a static list of submitted reports for use by audit resolution officials. We'll post more on this page if this becomes available. + ## March 18, 2024 **Audit search is currently limited to UEI-only.** You can search your UEI to confirm your audit submission is complete. The PDF report is still be available for download but summary report workbooks aren't. From 69c17fd3f435ee2c40c1250efeadf4862643d486 Mon Sep 17 00:00:00 2001 From: Laura Herring Date: Tue, 19 Mar 2024 15:46:14 -0400 Subject: [PATCH 6/6] status update copy --- src/status.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/src/status.md b/src/status.md index 9d0f44ef..205a188f 100644 --- a/src/status.md +++ b/src/status.md @@ -15,6 +15,8 @@ We're aware that search performance, even reduced to UEI-only, continues to be s We are also exploring options for using the API to generate a static list of submitted reports for use by audit resolution officials. We'll post more on this page if this becomes available. +For specific questions or issues, please contact the FAC via [our helpdesk](https://support.fac.gov/hc/en-us/requests/new). + ## March 18, 2024 **Audit search is currently limited to UEI-only.** You can search your UEI to confirm your audit submission is complete. The PDF report is still be available for download but summary report workbooks aren't.