Turn off Data Dashboard in production

[adamjtaylor]

our update to https://github.com/Sage-Bionetworks/data_curator/releases/tag/v22.11.3 turns on the data dashboard.

I imagine this will result in quite a few questions from centers about data completeness. I have some already.

I would like to explore the possibility of hiding the data daashboard in production until there is appropriate documentation for centers and we can properly launch the feature at a steering committee.

I would like to remove ASAP and then target adding back in during the 23/01 sprint.

[adamjtaylor]

Related issues:

• Data Dashboard points towards old manifests #49
• Data Dashboard causes disconnect for HTAN Duke project #48
• Data Dashboard causes disconnect for HTAN BU #52 (this seems to be different from the others)
• Data Dashboard causes disconnect for HTAN DFCI #53
• Data Dashboard causes disconnect for HTAN HMS #54
• Data Dashboard causes disconnect for HTAN HTAPP #55
• Data Dashboard causes disconnect for HTAN MSK #56
• Data Dashboard causes disconnect for HTAN TNP - TMA #57
• Data Dashboard causes disconnect for HTAN Vanderbilt #58
• Add documentation on using the Data Dashboard HTAN-Data-Ingress-Docs-ARCHIVED#31

[adamjtaylor]

FYI @milen-sage - we hare having some issues with the Data Dashboard in production.
Is it possible to selectively remove the Dashboard module to make sure we have the other fixes in data_curator v22.11.3 deployed, or are we best to revert to be level with data_curator v22.11.2 as per the draft PR linked.

When working the dashboard looks great but I think we should hold it to launch at a steering committee or similar once fixed, and update our FAQ docs on how to use/interpret at the same time

[adamjtaylor]

Confirming that we are going to revert per the PR above

[adamjtaylor]

• Merged
• Kicks off action that deploys this to staging https://github.com/ncihtan/HTAN-data-curator/actions/runs/3576731275
• Action successful
• Check that staging has removed the dashboard and seems to be functioning
• Make a draft release tagged v22.11.2. Note that this reverts us to DCA v22.11.2
• Wait for action to complete https://github.com/ncihtan/HTAN-data-curator/actions/runs/3576931102
• Confirm production looks OK
• PR to re-update to DCA v22.11.3
• Merge
• Confirm that this deploys in staging.
• Update staging to latest commit on main branch of data-models

[milen-sage]

@adamjtaylor I can confirm that reverting to codebase in the main branch of the DCA upstream (i.e. v22.11.2) would get the bug fixes noted in v22.11.3; you'd only be missing the dashboard features and suppression of warning messages in the log. The steps above are good to go!

[elv-sb]

@milen-sage we just completed the actions above. I can confirm that production looks ok again.

[milen-sage]

@adamjtaylor and @elv-sb could you start gh issues for the bugs/features you'd like to see with the dashboard (aside from docs - we are planning on that work)? Thanks!

[elv-sb]

@milen-sage we can add those during our next sprint. Thank you!

[milen-sage]

Thanks @elv-sb! Meanwhile feel free to explore more of the dashboard in the testing HTAN instance. Once we finish the docs on how schema validation rules are reflected in the dashboard and how the dashboard results can be interpreted we can have a dashboard exploration/play meeting. But certainly agreed with you and @adamjtaylor that we don't want to scare centers, if they see a bunch of their metadata, clinical and biospecimen data are marked as invalid out of the blue.

[adamjtaylor]

Bring to next data with FAIR on 12/12

What is the plan at the end of this sprint?

• Leave at DCA 22.11.2?
• Able to turn off data dashboard and use >=22.11.3
• Manually merge in bug fixes from 22.11.2 but not take the DCA commit. Not relevant. Only dashboard added between 11.2 and 11.3

[aclayton555]

Per discussion with Milen on 1/9 HTAN/FAIR stakeholder call:

Adam: Ability to toggle dashboard - is this being considered?
Milen: Approach will be to remove by default and make optional; only enable for projects that have gone through process of ramping up users/rolling this out - this will then be turned on. Next DCA release is expected to have this removed (targeted for 1/11; v23.1.1), so we can pull in latest release with other fixes

[afwillia]

@adamjtaylor the dashboard has been disabled by default in the main branch of DCA and release v23.1.1.

One note: you'll need to incorporate the changes in shinyapps_deploy.yml . The step Save service account credentials for Schematic will properly convert the schematic service account credentials stored as a github secret to a json file. Moving forward, schematic will authenticate the google API using the service account credentials instead of the token.pickle file.

[adamjtaylor]

I'd like to be able to update our DCA to v23.1.1, but when I try and open a PR for this it won't allow me
update-dca...Sage-Bionetworks:data_curator:v23.1.1

[adamjtaylor]

Following @milen-sage's comment in FAIR Data sprint review we should actually update to v23.1.2 (ie level with the head repo main branch) as the Google auth changes are included in the newest version.

[adamjtaylor]

Moving to ready for review as the Data Dashboard (Not DCA!) is off. Suggest we have a new issue for the cleanup and actually getting to the latest DCA version.

[milen-sage]

@adamjtaylor why is the DCA off? Based on discussion with @afwillia and the jira issue, I had assumed the DCA was running w/ updated schematic code (v23.1.1) and DCA v22.11.x, which would have switched to updated Google auth mechanism, even if not using the latest DCA version. I was assuming @afwillia was syncing up with @adamjtaylor to update the DCA to v23.1.3 to move on to the latest version but that was not a blocker for DCA running, based on conversation with @afwillia.

Was my understanding incorrect? Or is there a different issue?

[adamjtaylor]

@milen-sage Typo on my behalf. Data dashboard is currently off - not the DCA. Have updated comment.
Otherwise your understanding correct and we will need to sync up with @afwillia to get us fully up to v23.1.x
Sorry for the confusion!