Error with python prepare_C2M2_submission.py. Phenotype. HP:0025464 doesn't have genes associated with it

[mano-at-sdsc]

I am getting an error with:

python prepare_C2M2_submission.py

Looks like this is with Phenotype. HP:0025464 doesn't have genes associated with it, even though this term is in HPO obo file. Should I remove HP:0025464. Not sure this error might arise with other HPO terms as well.

Human Phenotype Ontology... [Tue Apr 12 11:07:54 PDT 2022]
Traceback (most recent call last):
File "prepare_C2M2_submission.py", line 1337, in
decorateTermsUsed()
File "prepare_C2M2_submission.py", line 430, in decorateTermsUsed
for entrez_id in hpo_to_entrez[currentTerm]:
KeyError: 'HP:0025464'

[raynamharris]

Indeed, HP:0025464 does't have any genes associated with it, so it is not present in hp.phenotype_to_genes.txt or HPO_Entrez_gene_IDs_to_EnsEMBL_IDs.tsv. But, who knows, maybe in the future it will be. Instead of deleting this entry or others like it, could we add NAs to the gene and ensemble id fields of those files to avoid the error?

[mano-at-sdsc]

I would love to see that I don't need to delete it from table and the issue be handled by the script prepare_C2M2_submission.py in conjunction with the C2M2 json schema.

[abradyIGS]

This is a script error. Fixing, stand by for an update to OSF

[abradyIGS]

@mano-at-sdsc This was caused by a bug due to a python block that was incorrectly indented. I've published a fixed version to https://osf.io/bq6k9/ -- please get the updated script and try again. The error should no longer occur.

[mano-at-sdsc]

Thanks a lot. It worked. After adding several header-only files, frictionless could validate all but one file.

phenotype_gene.tsv

error is like

invalid: phenotype_gene.tsv

-------

==== ===== ================= ==============================================================================
row field code message
==== ===== ================= ==============================================================================
921 foreign-key-error Row at position "921" violates the foreign key: not found in the lookup table
Several other lines like this

I checked line 921 is:
HP:0000501 ENSG00000288784

I checked HP:0000501 is listed in file phenotype.tsv [auto-generated], but gene ENSG00000288784 is not in file gene.tsv [auto-generated].

I had downloaded external ref zip file a few hours back. Please let me know where I might have left out something. Thanks.

[abradyIGS]

Progress! Investigating new thing, stand by.

[mano-at-sdsc]

Dear Arthur:

Any progress on this. If there an updated py script/json or external ref files that I should download and retry. Thanks.

[abradyIGS]

Sorry, got swamped by another request, nearly resolved here, stay tuned

[abradyIGS]

Found it. The link between the phenotype and the given gene is loading correctly, but the target gene isn't in my main Ensembl reference, which is a year old. Let me update and reprocess, shouldn't take too long, watch this space for updates

[abradyIGS]

@mano-at-sdsc Please re-download only ensembl_genes.tsv from here and rerun the script. Fingers crossed, this one should be taken care of: let me know how it goes.

[mano-at-sdsc]

Thanks. I can only imagine how busy you must be responding to questions from all DCCs.
Tried with newly downloaded ensembl_genes.tsv. It fixed 4 of them, but several still remain. Posting the list of lines with error:

invalid: phenotype_gene.tsv

row field code message
==== ===== ================= ==============================================================================
962 foreign-key-error Row at position "962" violates the foreign key: not found in the lookup table
1767 foreign-key-error Row at position "1767" violates the foreign key: not found in the lookup table
1770 foreign-key-error Row at position "1770" violates the foreign key: not found in the lookup table
1891 foreign-key-error Row at position "1891" violates the foreign key: not found in the lookup table
1977 foreign-key-error Row at position "1977" violates the foreign key: not found in the lookup table
1978 foreign-key-error Row at position "1978" violates the foreign key: not found in the lookup table
2417 foreign-key-error Row at position "2417" violates the foreign key: not found in the lookup table
2537 foreign-key-error Row at position "2537" violates the foreign key: not found in the lookup table
2804 foreign-key-error Row at position "2804" violates the foreign key: not found in the lookup table
2959 foreign-key-error Row at position "2959" violates the foreign key: not found in the lookup table
3167 foreign-key-error Row at position "3167" violates the foreign key: not found in the lookup table
3324 foreign-key-error Row at position "3324" violates the foreign key: not found in the lookup table
3421 foreign-key-error Row at position "3421" violates the foreign key: not found in the lookup table
3821 foreign-key-error Row at position "3821" violates the foreign key: not found in the lookup table
3827 foreign-key-error Row at position "3827" violates the foreign key: not found in the lookup table
4034 foreign-key-error Row at position "4034" violates the foreign key: not found in the lookup table
4211 foreign-key-error Row at position "4211" violates the foreign key: not found in the lookup table
4219 foreign-key-error Row at position "4219" violates the foreign key: not found in the lookup table
4293 foreign-key-error Row at position "4293" violates the foreign key: not found in the lookup table
4640 foreign-key-error Row at position "4640" violates the foreign key: not found in the lookup table
4934 foreign-key-error Row at position "4934" violates the foreign key: not found in the lookup table
4935 foreign-key-error Row at position "4935" violates the foreign key: not found in the lookup table
5417 foreign-key-error Row at position "5417" violates the foreign key: not found in the lookup table
5448 foreign-key-error Row at position "5448" violates the foreign key: not found in the lookup table
5736 foreign-key-error Row at position "5736" violates the foreign key: not found in the lookup table
6280 foreign-key-error Row at position "6280" violates the foreign key: not found in the lookup table
6292 foreign-key-error Row at position "6292" violates the foreign key: not found in the lookup table
6472 foreign-key-error Row at position "6472" violates the foreign key: not found in the lookup table
6958 foreign-key-error Row at position "6958" violates the foreign key: not found in the lookup table

If needed, you can look at the file phenotype_gene.tsv here: phenotype_gene.tsv

I also extracted the relevant lines from phenotype_gene.tsv (lists ENSG not found in ensembl_genes.tsv):
HP:0000501 ENSG00000264229
HP:0000717 ENSG00000259905
HP:0000717 ENSG00000201831
HP:0000822 ENSG00000238923
HP:0000822 ENSG00000130600
HP:0000822 ENSG00000288237
HP:0000964 ENSG00000264229
HP:0001369 ENSG00000244040
HP:0001369 ENSG00000208017
HP:0001513 ENSG00000214548
HP:0001513 ENSG00000201831
HP:0001513 ENSG00000235590
HP:0001513 ENSG00000259905
HP:0001596 ENSG00000269900
HP:0001596 ENSG00000264229
HP:0001638 ENSG00000264229
HP:0001638 ENSG00000214548
HP:0001638 ENSG00000269900
HP:0001638 ENSG00000269821
HP:0001943 ENSG00000269821
HP:0001943 ENSG00000130600
HP:0001943 ENSG00000288237
HP:0002758 ENSG00000208017
HP:0002829 ENSG00000244040
HP:0002829 ENSG00000264229
HP:0010535 ENSG00000259905
HP:0010535 ENSG00000201831
HP:0012622 ENSG00000238923
HP:0100806 ENSG00000269900

[abradyIGS]

Update: ENSG00000264229 (e.g.) clearly exists but does not appear in the reference gff3s for the GRCh38.106 assembly. I'm going to try to find whatever supplemental annotation includes it; if I can't, I'm just going to explicitly strip out access to any IDs not already in ensembl_genes.tsv and call it a day. Either way, this should solve this problem once and for all. Stand by and thanks again for your patience.

[abradyIGS]

@mano-at-sdsc :: Okay, try this -- re-download only HPO_Entrez_gene_IDs_to_EnsEMBL_IDs.tsv from the external_cv_reference_files subfolder here and re-run the script.

[mano-at-sdsc]

@abradyIGS Thanks. Awesome. Validated. Will proceed with submission.

[abradyIGS]

Incidentally @mano-at-sdsc I just updated the main gene ref with the big alias file you sent a few weeks ago (sorry: it dropped off my radar until early this morning).

i recommend getting the new ensembl_genes.tsv i just uploaded here and rebuilding your submission: the result will have all the available HGNC, RefSeq, Entrez, MIM, etc. xrefs as synonyms.

[mano-at-sdsc]

@abradyIGS The file ensembl_genes.tsv is just 1.1kb and seems to have binary characters. Please check and let me know.

[abradyIGS]

MacOS alias/upload error, sorry -- replaced just now, please try again. Correct size should be 9,377,130 bytes.

[mano-at-sdsc]

Tried. Getting this error when trying to validate with frictionless in gene.tsv:

-------

invalid: gene.tsv

-------

==== ===== ==========
row field code message
==== ===== ==========
7 4 type-error Type error in the cell "["ENTREZ:5893", "HGNC:9824", "MIM:600392", "NM_001297419", "NM_001297420", "NM_001297421", "NM_001297422", "NM_134422", "NM_134423", "NM_134424", "NP_001284348", "NP_001284349", "NP_001284350", "NP_001284351", "NP_602296", "NR_123713", "RAD52 homolog, "RAD52", "XM_005253720", "XM_005253721", "XM_011520990", "XM_011520991", "XM_011520995", "XM_017019769", "XM_017019770", "XM_017019771", "XP_005253777", "XP_005253778", "XP_011519292", "XP_011519293", "XP_011519297", "XP_016875258", "XP_016875259", "XP_016875260", "XR_001748836", "XR_001748837", "XR_001748838", "XR_001748839", "XR_002957366", "XR_002957367", "XR_931522", DNA repair protein"]" in row "7" and field "synonyms" at position "4": type is "array/default"
several rows like this.

[abradyIGS]

Thanks -- I see the problem, fixing, stand by

Sorry, lot of hotfixes this week

[abradyIGS]

Have pushed an updated copy (v5, now). Please retry and let me know ASAP if any other errors are encountered.

[mano-at-sdsc]

Thanks. Validated and have submitted. Will review on the portal once digested. I assume this would not have affected any other sections such as files, biosample, etc., which I had reviewed for yesterday's submission.

[abradyIGS]

Assumption correct -- glad to hear it worked at last.

[mano-at-sdsc]

@abradyIGS Thanks a lot for fixing these things instantly. Submission seems to have gone through just fine, reviewed several things randomly. For gene search, on the portal (content yet not approved), it doesn't show other ID types, e.g., Entrez ID or REFSEQ, but I suppose that is just the current display that will change in the future since all the information is captured in gene.tsv thanks to all your hard work.

[mano-at-sdsc]

Actually it was able to search by synonym [e.g., ABCC7], https://app.nih-cfde.org/chaise/record/#121/CFDE:gene/nid=5, so it is all there.