Data Dashboard causes disconnect for HTAN BU

[adamjtaylor]

On opening Data Dashboard for HTAN BU project get a disconnect within a few seconds

2022-11-29T10:15:13.294523+00:00 shinyapps[5806160]: WARNING: [2022-11-29 10:15:13] root - For the attribute 'Environmental Tobacco Smoke Exposure', on row 4, 'unknown' is not one of ['No', 'Yes', 'Unknown', ''].
2022-11-29T10:15:13.294571+00:00 shinyapps[5806160]: WARNING: [2022-11-29 10:15:13] root - For the attribute 'Alcohol Type', on row 5, 'unknown' is not one of ['Wine', 'Beer', 'Other', 'Liquor', 'Not Reported', 'Unknown', ''].
2022-11-29T10:15:13.295641+00:00 shinyapps[5806160]: WARNING: [2022-11-29 10:15:13] root - For the attribute 'Alcohol Intensity', on row 5, 'unknown' is not one of ['Drinker', 'Non-Drinker', 'Occasional Drinker', 'Lifelong Non-Drinker', 'Heavy Drinker', 'Not Reported', 'Unknown', ''].
2022-11-29T10:15:13.295675+00:00 shinyapps[5806160]: WARNING: [2022-11-29 10:15:13] root - For the attribute 'Alcohol Type', on row 5, 'unknown' is not one of ['Wine', 'Beer', 'Other', 'Liquor', 'Not Reported', 'Unknown', ''].
2022-11-29T10:15:13.296095+00:00 shinyapps[5806160]: WARNING: [2022-11-29 10:15:13] root - For the attribute 'Alcohol Intensity', on row 5, 'unknown' is not one of ['Drinker', 'Non-Drinker', 'Occasional Drinker', 'Lifelong Non-Drinker', 'Heavy Drinker', 'Not Reported', 'Unknown', ''].
2022-11-29T10:15:13.296138+00:00 shinyapps[5806160]: WARNING: [2022-11-29 10:15:13] root - For the attribute 'Alcohol Type', on row 7, 'unknown' is not one of ['Wine', 'Beer', 'Other', 'Liquor', 'Not Reported', 'Unknown', ''].
2022-11-29T10:15:13.296180+00:00 shinyapps[5806160]: WARNING: [2022-11-29 10:15:13] root - For the attribute 'Alcohol Intensity', on row 7, 'unknown' is not one of ['Drinker', 'Non-Drinker', 'Occasional Drinker', 'Lifelong Non-Drinker', 'Heavy Drinker', 'Not Reported', 'Unknown', ''].
2022-11-29T10:15:13.296233+00:00 shinyapps[5806160]: WARNING: [2022-11-29 10:15:13] root - For the attribute 'Alcohol Type', on row 7, 'unknown' is not one of ['Wine', 'Beer', 'Other', 'Liquor', 'Not Reported', 'Unknown', ''].
2022-11-29T10:15:13.296279+00:00 shinyapps[5806160]: WARNING: [2022-11-29 10:15:13] root - For the attribute 'Alcohol Intensity', on row 7, 'unknown' is not one of ['Drinker', 'Non-Drinker', 'Occasional Drinker', 'Lifelong Non-Drinker', 'Heavy Drinker', 'Not Reported', 'Unknown', ''].
2022-11-29T10:15:13.694484+00:00 shinyapps[5806160]: WARNING: [2022-11-29 10:15:13] root - For the attribute 'Treatment Effect', on row 5, 'unknown' is not one of ['Incomplete Necrosis (Viable Tumor Present)', 'No Necrosis', 'Complete Necrosis (No Viable Tumor)', 'Not Reported', 'Unknown', ''].
2022-11-29T10:15:13.694533+00:00 shinyapps[5806160]: WARNING: [2022-11-29 10:15:13] root - For the attribute 'Treatment Effect Indicator', on row 5, 'unknown' is not one of ['No', 'Yes', 'Not Reported', 'Unknown', ''].
2022-11-29T10:15:13.798034+00:00 shinyapps[5806160]: WARNING: [2022-11-29 10:15:13] root - For the attribute 'COVID19 Antibody Testing', on row 2, 'unknown' is not one of ['SARS-CoV-2 IgG antibody positive', 'SARS-CoV-2 IgG/IgM antibody negative', 'SARS-CoV-2 IgG/IgM antibody positive', 'SARS-CoV-2 IgG antibody negative', 'Not Done', 'SARS-CoV-2 IgM antibody negative', 'SARS-CoV-2 IgM antibody positive', 'Not Applicable', 'Unknown', ''].
2022-11-29T10:15:13.798072+00:00 shinyapps[5806160]: WARNING: [2022-11-29 10:15:13] root - For the attribute 'Ecig vape use', on row 2, 'unknown' is not one of ['Former', 'Current', 'Never', 'Unknown', ''].
2022-11-29T10:15:13.798765+00:00 shinyapps[5806160]: WARNING: [2022-11-29 10:15:13] root - For the attribute 'COVID19 Antibody Testing', on row 3, 'unknown' is not one of ['SARS-CoV-2 IgG antibody positive', 'SARS-CoV-2 IgG/IgM antibody negative', 'SARS-CoV-2 IgG/IgM antibody positive', 'SARS-CoV-2 IgG antibody negative', 'Not Done', 'SARS-CoV-2 IgM antibody negative', 'SARS-CoV-2 IgM antibody positive', 'Not Applicable', 'Unknown', ''].
2022-11-29T10:15:13.798782+00:00 shinyapps[5806160]: WARNING: [2022-11-29 10:15:13] root - For the attribute 'Family Member Vital Status Indicator', on row 3, 'unknown' is not one of ['Vital Status Alive', 'Unknown', 'Not Reported', 'Vital Status Dead', ''].
2022-11-29T10:15:13.799017+00:00 shinyapps[5806160]: WARNING: [2022-11-29 10:15:13] root - For the attribute 'Ecig vape use', on row 3, 'unknown' is not one of ['Former', 'Current', 'Never', 'Unknown', ''].
2022-11-29T10:15:13.799348+00:00 shinyapps[5806160]: WARNING: [2022-11-29 10:15:13] root - For the attribute 'COVID19 Antibody Testing', on row 4, 'unknown' is not one of ['SARS-CoV-2 IgG antibody positive', 'SARS-CoV-2 IgG/IgM antibody negative', 'SARS-CoV-2 IgG/IgM antibody positive', 'SARS-CoV-2 IgG antibody negative', 'Not Done', 'SARS-CoV-2 IgM antibody negative', 'SARS-CoV-2 IgM antibody positive', 'Not Applicable', 'Unknown', ''].
2022-11-29T10:15:13.799421+00:00 shinyapps[5806160]: WARNING: [2022-11-29 10:15:13] root - For the attribute 'Family Member Vital Status Indicator', on row 4, 'unknown' is not one of ['Vital Status Alive', 'Unknown', 'Not Reported', 'Vital Status Dead', ''].
2022-11-29T10:15:13.799460+00:00 shinyapps[5806160]: WARNING: [2022-11-29 10:15:13] root - For the attribute 'Ecig vape use', on row 4, 'unknown' is not one of ['Former', 'Current', 'Never', 'Unknown', ''].
2022-11-29T10:15:13.799965+00:00 shinyapps[5806160]: WARNING: [2022-11-29 10:15:13] root - For the attribute 'COVID19 Antibody Testing', on row 5, 'unknown' is not one of ['SARS-CoV-2 IgG antibody positive', 'SARS-CoV-2 IgG/IgM antibody negative', 'SARS-CoV-2 IgG/IgM antibody positive', 'SARS-CoV-2 IgG antibody negative', 'Not Done', 'SARS-CoV-2 IgM antibody negative', 'SARS-CoV-2 IgM antibody positive', 'Not Applicable', 'Unknown', ''].
2022-11-29T10:15:13.800061+00:00 shinyapps[5806160]: WARNING: [2022-11-29 10:15:13] root - For the attribute 'Ecig vape use', on row 5, 'unknown' is not one of ['Former', 'Current', 'Never', 'Unknown', ''].
2022-11-29T10:15:13.801469+00:00 shinyapps[5806160]: WARNING: [2022-11-29 10:15:13] root - For the attribute 'COVID19 Antibody Testing', on row 6, 'unknown' is not one of ['SARS-CoV-2 IgG antibody positive', 'SARS-CoV-2 IgG/IgM antibody negative', 'SARS-CoV-2 IgG/IgM antibody positive', 'SARS-CoV-2 IgG antibody negative', 'Not Done', 'SARS-CoV-2 IgM antibody negative', 'SARS-CoV-2 IgM antibody positive', 'Not Applicable', 'Unknown', ''].
2022-11-29T10:15:13.801506+00:00 shinyapps[5806160]: WARNING: [2022-11-29 10:15:13] root - For the attribute 'Ecig vape use', on row 6, 'unknown' is not one of ['Former', 'Current', 'Never', 'Unknown', ''].
2022-11-29T10:15:13.801511+00:00 shinyapps[5806160]: WARNING: [2022-11-29 10:15:13] root - For the attribute 'COVID19 Antibody Testing', on row 7, 'unknown' is not one of ['SARS-CoV-2 IgG antibody positive', 'SARS-CoV-2 IgG/IgM antibody negative', 'SARS-CoV-2 IgG/IgM antibody positive', 'SARS-CoV-2 IgG antibody negative', 'Not Done', 'SARS-CoV-2 IgM antibody negative', 'SARS-CoV-2 IgM antibody positive', 'Not Applicable', 'Unknown', ''].
2022-11-29T10:15:13.801516+00:00 shinyapps[5806160]: WARNING: [2022-11-29 10:15:13] root - For the attribute 'Family Member Vital Status Indicator', on row 7, 'unknown' is not one of ['Vital Status Alive', 'Unknown', 'Not Reported', 'Vital Status Dead', ''].
2022-11-29T10:15:13.801519+00:00 shinyapps[5806160]: WARNING: [2022-11-29 10:15:13] root - For the attribute 'Ecig vape use', on row 7, 'unknown' is not one of ['Former', 'Current', 'Never', 'Unknown', ''].
2022-11-29T10:15:14.207174+00:00 shinyapps[5806160]: Warning in value[[3L]](cond) :
2022-11-29T10:15:14.207855+00:00 shinyapps[5806160]:   'validateModelManifest' failed: ‘syn24986734’:
2022-11-29T10:15:14.208297+00:00 shinyapps[5806160]: KeyError: ''
2022-11-29T10:15:14.209081+00:00 shinyapps[5806160]: Detailed traceback:
2022-11-29T10:15:14.209507+00:00 shinyapps[5806160]:   File "/srv/connect/apps/HTAN-data-curator/.venv/lib/python3.8/site-packages/schematic/models/metadata.py", line 254, in validateModelManifest
2022-11-29T10:15:14.209909+00:00 shinyapps[5806160]:     errors, warnings, manifest = validate_all(self, errors, warnings, manifest, manifestPath, self.sg, jsonSchema, restrict_rules, project_scope)
2022-11-29T10:15:14.210374+00:00 shinyapps[5806160]:   File "/srv/connect/apps/HTAN-data-curator/.venv/lib/python3.8/site-packages/schematic/models/validate_manifest.py", line 259, in validate_all
2022-11-29T10:15:14.210769+00:00 shinyapps[5806160]:     vmv_errors, vmv_warnings = vm.validate_manifest_values(manifest, jsonSchema, sg)
2022-11-29T10:15:14.211129+00:00 shinyapps[5806160]:   File "/srv/connect/apps/HTAN-data-curator/.venv/lib/python3.8/site-packages/schematic/models/validate_manifest.py", line 241, in validate_manifest_values
2022-11-29T10:15:14.211482+00:00 shinyapps[5806160]:     val_errors, val_warnings =  GenerateError.generate_schema_error(row_num = errorRow, attribute_name = errorColName, error_msg = errorMsg, invalid_entry = errorVal, sg = sg)
2022-11-29T10:15:14.211838+00:00 shinyapps[5806160]:   File "/srv/connect/apps/HTAN-data-curator/.venv/lib/python3.8/ [... truncated]
2022-11-29T10:15:14.950442+00:00 shinyapps[5806160]: Warning in parallel::mclapply(1:nrow(metadata), function(i) { :
2022-11-29T10:15:14.950479+00:00 shinyapps[5806160]:   scheduled cores 7, 5, 6, 12, 10 encountered errors in user code, all values of the jobs will be affected
2022-11-29T10:15:15.056282+00:00 shinyapps[5806160]: Warning: Error in bind_rows: Argument 5 must have names.
2022-11-29T10:15:15.069539+00:00 shinyapps[5806160]:   15: <Anonymous>
2022-11-29T10:15:15.069570+00:00 shinyapps[5806160]:   13: fn
2022-11-29T10:15:15.069575+00:00 shinyapps[5806160]:    8: retry
2022-11-29T10:15:15.069578+00:00 shinyapps[5806160]:    7: connect$retryingStartServer
2022-11-29T10:15:15.069582+00:00 shinyapps[5806160]:    6: eval
2022-11-29T10:15:15.069585+00:00 shinyapps[5806160]:    5: eval
2022-11-29T10:15:15.069598+00:00 shinyapps[5806160]:    4: eval
2022-11-29T10:15:15.069601+00:00 shinyapps[5806160]:    3: eval
2022-11-29T10:15:15.069604+00:00 shinyapps[5806160]:    2: eval.parent
2022-11-29T10:15:15.069608+00:00 shinyapps[5806160]:    1: local

[aclayton555]

Included in escalation to FAIR in this issue: https://sagebionetworks.jira.com/servicedesk/customer/portal/5/IBCSPRT-115

[rrchai]

@milen-sage Based on the test in #63:

I think it might be a bug from the backend. Here is what might cause the error.

The dashboard is expecting "errors" and "warnings" two outputs from the validation results, where "warnings" should return the concatenated results for cross-manifest check. However, using the latest develop of schematic, one(s) of manifests have invalid errors from "warnings" output, which makes dashboard function fail to process the results. For example:

synapse get syn24994554  
schematic model -c schematic_config.yml validate -mp synapse_storage_manifest.csv -dt Diagnosis -rr

warning: For the attribute 'AJCC Clinical T', on row 4, 1670472942462 is not one of ['T2a1', 'T3b', 'T1b1', 'T2a2', 'T1', 'T2b', 'T2', 'Ta', "Tis (Paget's)", 'Tis', 'T2a', 'TX', 'T1a1', 'T2c', 'T4a', 'T1a2', 'T2d', 'T4e', 'Tis (LCIS)', 'T0', 'T1c', 'T3', 'T3d', 'T4b', 'T4', 'Tis (DCIS)', 'unknown', 'T1mi', 'Not Reported', 'T3c', 'T3a', 'T1a', 'T4d', 'T4c', 'T1b', 'T1b2', ''].
warning: For the attribute 'AJCC Pathologic T', on row 4, 1670472942789 is not one of ['T2a1', 'T3b', 'T1b1', 'T2a2', 'T1', 'T2b', 'T2', 'Ta', "Tis (Paget's)", 'Tis', 'T2a', 'TX', 'T1a1', 'T2c', 'T4a', 'T1a2', 'T2d', 'T4e', 'Tis (LCIS)', 'T0', 'T1c', 'T3', 'T3d', 'T4b', 'T4', 'Tis (DCIS)', 'unknown', 'T1mi', 'Not Reported', 'T3c', 'T3a', 'T1a', 'T4d', 'T4c', 'T1b', 'T1b2', ''].
warning: For the attribute 'AJCC Clinical T', on row 6, 1667884542480 is not one of ['T2a1', 'T3b', 'T1b1', 'T2a2', 'T1', 'T2b', 'T2', 'Ta', "Tis (Paget's)", 'Tis', 'T2a', 'TX', 'T1a1', 'T2c', 'T4a', 'T1a2', 'T2d', 'T4e', 'Tis (LCIS)', 'T0', 'T1c', 'T3', 'T3d', 'T4b', 'T4', 'Tis (DCIS)', 'unknown', 'T1mi', 'Not Reported', 'T3c', 'T3a', 'T1a', 'T4d', 'T4c', 'T1b', 'T1b2', ''].
warning: For the attribute 'AJCC Pathologic T', on row 6, 1667884542807 is not one of ['T2a1', 'T3b', 'T1b1', 'T2a2', 'T1', 'T2b', 'T2', 'Ta', "Tis (Paget's)", 'Tis', 'T2a', 'TX', 'T1a1', 'T2c', 'T4a', 'T1a2', 'T2d', 'T4e', 'Tis (LCIS)', 'T0', 'T1c', 'T3', 'T3d', 'T4b', 'T4', 'Tis (DCIS)', 'unknown', 'T1mi', 'Not Reported', 'T3c', 'T3a', 'T1a', 'T4d', 'T4c', 'T1b', 'T1b2', ''].

[milen-sage]

Thanks @rrchai ! @GiaJordan we can add this to next sprint?

[rrchai]

@milen-sage Feel free to tag me if the ^ issue is solved from the backend - I can test it again.

[GiaJordan]

The behavior Rong described above is intentional and was new as of PR #1000 in v22.11.3.
@afwillia will update the dashboard to accept the new schematic output