Skip to content

Commit

Permalink
Suggestions for increased transparency (#95)
Browse files Browse the repository at this point in the history
* Suggestions for increased transparency

This PR suggests some text changes to the copy, and adds back a chronology.

Because our site is used by:

1. Auditors and auditees submitting audits
2. State auditors (and other entities) responsible for passthrough monitoring
3. Federal audit resolution officials
4. IG offices
5. Grants managers/grants issuing offices
6. GAO
8. OMB
9. The general public
10. ...

I think credibility is enhanced by transparency, and because we have a
public record of our efforts in this space (our Github repository), I
think we should use it to document the actions we've taken so far.

For consideration.

* Simplification in second para.

* One more.

* copy edits to status page

* march 19 status update

* status update copy

---------

Co-authored-by: Laura Herring <[email protected]>
  • Loading branch information
jadudm and lauraherring authored Mar 20, 2024
1 parent f78c551 commit 4d8df44
Showing 1 changed file with 39 additions and 6 deletions.
45 changes: 39 additions & 6 deletions src/status.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,13 +5,46 @@ meta:
name: FAC system status
description: Track the operating status of the FAC, both for audit submission and search.
---
# FAC system status

# March 18, 2024
This page tracks the operating status of the FAC, both for audit submission and search.

**Audit search is currently limited to UEI-only.** You can search your UEI to confirm your audit submission is complete. The PDF report will still be available for download but summary report workbooks won't.
## March 19

## Issue background
In mid-February, GSA completed the migration of historical data from the Census Bureau. This increased the amount of data in the GSA system exponentially and slowed down search performance.
Our team has released multiple patches to stabilize search performance since then but have been unable to do so consistently. For this reason, we've decided to reduce search to UEI-only while we focus on building new search tables instead of one-off bug fixes.
We're aware that search performance, even reduced to UEI-only, continues to be slow. Our engineers are investigating the root cause of the issue and continue to work on a solution.

We're working as quickly and efficiently as possible to return full search functionality to our users as soon as we can. Thank you for your patience.
We are also exploring options for using the API to generate a static list of submitted reports for use by audit resolution officials. We'll post more on this page if this becomes available.

For specific questions or issues, please contact the FAC via [our helpdesk](https://support.fac.gov/hc/en-us/requests/new).

## March 18, 2024

**Audit search is currently limited to UEI-only.** You can search your UEI to confirm your audit submission is complete. The PDF report is still be available for download but summary report workbooks aren't.

### Issue background

In mid-February, GSA completed the migration of historical data from the Census Bureau. This increased the amount of data in the GSA system exponentially, hindering search performance.

Over the last month, our team has released multiple patches to stabilize search with limited success. Because of this, we've reduced search to UEI-only while we focus on building a more sustainable search function.

We're working hard to return full search functionality as soon as we can. We know this is a significant disruption for many users. Thank you for your patience.

**A timeline of the issues:**

- February 7: We completed the data migration. This increased the number of audits from 10K to more than 250K, and the number of awards tracked from 100K to nearly 4.5M.

- February 8: We [increased resources in our production environment](https://github.com/GSA-TTS/FAC/pull/3376) to address the increased load.

- February 9: The team removed the [My ALN filter](https://github.com/GSA-TTS/FAC/pull/3378) in an attempt to stop system crashes caused by this feature.

- February 14: We worked with more than a dozen NSACs to [prioritize search needs](https://github.com/GSA-TTS/FAC/issues/3388).

- February 16: The team [increased search capacity in the production environment](https://github.com/GSA-TTS/FAC/pull/3424).

- February 28: We [added prioritized search needs](https://github.com/GSA-TTS/FAC/pull/3440), re-introducing performance challenges.

- March 11: Heavy use of the system prevented system updates for nearly a week. The team completed [an after-hours update to the production environment](https://github.com/GSA-TTS/FAC/pull/3496), partially addressing new search performance issues.

- March 14: Performance issues and system crashes persisted in search. The team released [new search tables and queries](https://github.com/GSA-TTS/FAC/pull/3511). These required significant engineering and testing work.

- March 18: The team decided to take down search to focus on long-term solutions vs. short-term patches.

0 comments on commit 4d8df44

Please sign in to comment.