Update build procedure of imports

[allenbaron]

This PR accomplishes the following:

1. Restores the previous (indirect) import of SYMP & TRANS while retaining all term annotations in the source files including definitions.
   • Only deprecated terms and ontology annotations are removed from the source files.
2. Adds the versionIRI of import sources to their derivative <import name>_import.owl files as an rdfs:comment at the time a <import name>_import.owl file is built.
3. Updates the versionIRI of all imports each release, in the same way the versionIRI doid.owl is updated.
   • This is necessary for true versioning of import modules and ensures that <import_name>_import.owl files are accessible online by both their ontology IRI and version IRIs.
4. Adds make refresh_<import_name> commands to allow re-downloading a specific import source when re-building that import (as opposed to redownloading all imports OR rebuilding the import from a previously downloaded source file, when it exists).

Note that make clean_imports has also been renamed to make refresh_imports to better reflect its use in re-downloading AND rebuilding imports.

[allenbaron]

We should test the following to make sure these updates work as intended before merging this into main for production:

1. Imports build correctly.
   • Test with the latest release of SYMP:
     • The new terms are added and appear as expected.
     • No term annotations (definitions, ids, labels, xrefs, etc) are removed.
     • All deprecated terms are removed.
     • The ontology annotation information is updated.
       • The ontology IRI should be http://purl.obolibrary.org/obo/doid/imports/symp_import.owl
       • The version IRI at the time symp_import.owl is first built is the same as the ontology IRI.
       • An rdfs:comment is added including the version IRI of the symp.owl source file. Should be "Source from http://purl.obolibrary.org/obo/symp/releases/2022-07-05/symp.owl".
   • Test with the latest release of NCBItaxon (using NCBItaxon term updates listed in Terms to add to imports #946):
     • The new terms are added and appear as expected.
     • The types of annotations are the same (id, label, deprecated, definition).
     • The resulting ncbitaxon_import.owl has an rdfs:comment including the version IRI of the ncbitaxon.owl source file and IRIs remain the same.
2. make refresh_<import_name> works as expected (test with symp after tests in 1).
   • The build/symp.owl file re-downloads and symp_import.owl file is re-built (file contents remain the same).
3. Imports are properly versioned during the build of a release.
   • Test this by executing a test release on this branch.
   • All imports now have versionIRIs matching http://purl.obolibrary.org/obo/doid/releases/<date>/imports/<import name>_import.owl and are otherwise unchanged.
   • ext.owl has a versionIRI matching http://purl.obolibrary.org/obo/doid/releases/<date>/ext.owl and is otherwise unchanged.

[jbmunro]

1. Test with the latest release of SYMP = done.
2. make refresh_symp = works as expected.

[allenbaron]

ncbitaxon_import.owl built correctly but there are some old terms that were removed. When I resolve #1053 I'll refresh the import and commit it but that work doesn't need to hold up this PR.

[allenbaron]

Fixed errors in FOODON and CL by updating. Had to modify build commands to remove extra branches in FOODON and to avoid extra terms from outside both ontologies namespaces (BFO + CARO for CL and BFO + PO for FOODON).

We should consider switching sources to ontology -base.owl files where they are produced. This could simplify some of our import build commands.

[lschriml]

agreed, let's look into the switch for the August build.

…

On Wed, Jul 27, 2022 at 10:13 AM J. Allen Baron ***@***.***> wrote: Fixed errors in FOODON and CL by updating. Had to modify build commands to remove extra branches in FOODON and to avoid extra terms from outside both ontologies namespaces (BFO + CARO for CL and BFO + PO for FOODON). We should consider switching sources to ontology -base.owl files where they are produced. This could simplify some of our import build commands. — Reply to this email directly, view it on GitHub <#1052 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABBB4DM3T7K6IS374MHRXNTVWE7XVANCNFSM53H645VQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>

-- Lynn M. Schriml, Ph.D. Associate Professor Institute for Genome Sciences University of Maryland School of Medicine Department of Epidemiology and Public Health 670 W. Baltimore St., HSFIII, Room 3061 Baltimore, MD 21201 P: 410-706-6776 | F: 410-706-6756 ***@***.***

[allenbaron]

I went ahead and refreshed all the imports, excluding ECO, DISDRIV, and OMIM_SUSC which don't have automated build rules yet, to ensure they are all up-to-date and formatted as desired (with or without definitions).

For each updated import, I compared the previous version with the updated version 1) with Protege to verify the trees look as expected and their are no unexpected branches and 2) with robot diff. I also checked doid-edit.owl after each import was updated to ensure it looked okay.

All of these updates appear correct to me.

[allenbaron]

Finally, I did a test merge with the main branch and ran make release. There were no errors in the validation tests and only minor errors in building the obo files which I corrected (duplicate labels due to 2 SO terms and 1 UBERON term having labels in ext.owl). The release then completed successfully.

I then reviewed ext.owl and all of the imports (again) and they appeared to have correctly updated versionIRIs and otherwise looked as expected.

Finally, I reviewed all the reports. There are a few warnings that may be worth addressing (I did fix 1 whitespace warning) but nothing breaking.

This PR is now ready to be merged.