Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Remove absolute paths and redundant file references #13

Open
benjaminhwilliams opened this issue Apr 1, 2020 · 1 comment
Open

Remove absolute paths and redundant file references #13

benjaminhwilliams opened this issue Apr 1, 2020 · 1 comment

Comments

@benjaminhwilliams
Copy link
Member

  • Several experiment list files contain references to image file templates and/or mask files that refer to an absolute path.
  • In some cases, the file name doesn't match the example image/mask files in the same directory, presumably the intended target of these references.
  • In some cases, there is no matching file uploaded.

The latter two points aren't necessarily a problem until/unless someone writes a test that queries the linked mask/image file(s), but it seems sensible to make the uploaded data consistent and complete where possible, to avoid future test-writers getting caught out.

Examples:

$ grep -r "Users\|home"
l_cysteine_integrated/30_integrated_experiments.json:      "template": "/Users/graeme/data/i19-1/l-cyst/l-cyst_02_#####.cbf", 
l_cysteine_integrated/30_integrated_experiments.json:      "mask": "/Users/graeme/Projects/l-cys/DEFAULT/NATIVE/SWEEP1/index/hot_mask_0.pickle", 
l_cysteine_integrated/25_integrated_experiments.json:      "template": "/Users/graeme/data/i19-1/l-cyst/l-cyst_01_#####.cbf", 
l_cysteine_integrated/25_integrated_experiments.json:      "mask": "/Users/graeme/Projects/l-cys/DEFAULT/NATIVE/SWEEP1/index/hot_mask_0.pickle", 
l_cysteine_integrated/35_integrated_experiments.json:      "template": "/Users/graeme/data/i19-1/l-cyst/l-cyst_03_#####.cbf", 
l_cysteine_integrated/35_integrated_experiments.json:      "mask": "/Users/graeme/Projects/l-cys/DEFAULT/NATIVE/SWEEP1/index/hot_mask_0.pickle", 
l_cysteine_integrated/20_integrated_experiments.json:      "template": "/Users/graeme/data/i19-1/l-cyst/l-cyst_04_#####.cbf", 
l_cysteine_integrated/20_integrated_experiments.json:      "mask": "/Users/graeme/Projects/l-cys/DEFAULT/NATIVE/SWEEP1/index/hot_mask_0.pickle", 
pdteet-2-lattice/sweep-04/indexed.expt:      "template": "/Users/graeme/data/i19-1-double-PdTEET/PdTEET_04_#####.cbf.gz", 
l_cysteine_4_sweeps_scaled/scaled_20_25.expt:      "template": "/Users/graeme/data/i19-1/l-cyst/l-cyst_04_#####.cbf", 
l_cysteine_4_sweeps_scaled/scaled_20_25.expt:      "mask": "/Users/graeme/Projects/l-cys/DEFAULT/NATIVE/SWEEP1/index/hot_mask_0.pickle", 
l_cysteine_4_sweeps_scaled/scaled_20_25.expt:      "template": "/Users/graeme/data/i19-1/l-cyst/l-cyst_01_#####.cbf", 
l_cysteine_4_sweeps_scaled/scaled_20_25.expt:      "mask": "/Users/graeme/Projects/l-cys/DEFAULT/NATIVE/SWEEP1/index/hot_mask_0.pickle", 
l_cysteine_4_sweeps_scaled/scaled_30.expt:      "template": "/Users/graeme/data/i19-1/l-cyst/l-cyst_02_#####.cbf", 
l_cysteine_4_sweeps_scaled/scaled_30.expt:      "mask": "/Users/graeme/Projects/l-cys/DEFAULT/NATIVE/SWEEP1/index/hot_mask_0.pickle", 
l_cysteine_4_sweeps_scaled/scaled_35.expt:      "template": "/Users/graeme/data/i19-1/l-cyst/l-cyst_03_#####.cbf", 
l_cysteine_4_sweeps_scaled/scaled_35.expt:      "mask": "/Users/graeme/Projects/l-cys/DEFAULT/NATIVE/SWEEP1/index/hot_mask_0.pickle", 
centroid_test_data/INTEGRATE.HKL:!NAME_TEMPLATE_OF_DATA_FRAMES=/Users/graeme/data/RU/1/X4_lots_M1S4_1_0???.cbf CB
@benjaminhwilliams
Copy link
Member Author

A slightly more complete list:

$ grep -r "/dls/\|/Users/"
l_cysteine_integrated/30_integrated_experiments.json:      "template": "/Users/graeme/data/i19-1/l-cyst/l-cyst_02_#####.cbf", 
l_cysteine_integrated/30_integrated_experiments.json:      "mask": "/Users/graeme/Projects/l-cys/DEFAULT/NATIVE/SWEEP1/index/hot_mask_0.pickle", 
l_cysteine_integrated/25_integrated_experiments.json:      "template": "/Users/graeme/data/i19-1/l-cyst/l-cyst_01_#####.cbf", 
l_cysteine_integrated/25_integrated_experiments.json:      "mask": "/Users/graeme/Projects/l-cys/DEFAULT/NATIVE/SWEEP1/index/hot_mask_0.pickle", 
l_cysteine_integrated/35_integrated_experiments.json:      "template": "/Users/graeme/data/i19-1/l-cyst/l-cyst_03_#####.cbf", 
l_cysteine_integrated/35_integrated_experiments.json:      "mask": "/Users/graeme/Projects/l-cys/DEFAULT/NATIVE/SWEEP1/index/hot_mask_0.pickle", 
l_cysteine_integrated/20_integrated_experiments.json:      "template": "/Users/graeme/data/i19-1/l-cyst/l-cyst_04_#####.cbf", 
l_cysteine_integrated/20_integrated_experiments.json:      "mask": "/Users/graeme/Projects/l-cys/DEFAULT/NATIVE/SWEEP1/index/hot_mask_0.pickle", 
vmxi_proteinase_k_sweeps/experiments_3.expt:      "template": "/dls/mx/data/mx21314/mx21314-30/VMXi-AB0813/well_46/images/image_17657_master.h5", 
vmxi_proteinase_k_sweeps/experiments_3.expt:      "mask": "/dls/science/users/hko55533/mx21314-30/VMXi-AB0813/well_46_image_17657/DEFAULT/NATIVE/SWEEP1/index/2_hot_mask_0.pickle", 
vmxi_proteinase_k_sweeps/experiments_0.expt:      "template": "/dls/mx/data/mx21314/mx21314-30/VMXi-AB0813/well_112/images/image_17578_master.h5", 
vmxi_proteinase_k_sweeps/experiments_0.expt:      "mask": "/dls/science/users/hko55533/mx21314-30/VMXi-AB0813/well_112_image_17578/DEFAULT/NATIVE/SWEEP1/index/2_hot_mask_0.pickle", 
vmxi_proteinase_k_sweeps/experiments_1.expt:      "template": "/dls/mx/data/mx21314/mx21314-30/VMXi-AB0813/well_12/images/image_17586_master.h5", 
vmxi_proteinase_k_sweeps/experiments_1.expt:      "mask": "/dls/science/users/hko55533/mx21314-30/VMXi-AB0813/well_12_image_17586/DEFAULT/NATIVE/SWEEP1/index/2_hot_mask_0.pickle", 
vmxi_proteinase_k_sweeps/experiments_2.expt:      "template": "/dls/mx/data/mx21314/mx21314-30/VMXi-AB0813/well_151/images/image_17607_master.h5", 
vmxi_proteinase_k_sweeps/experiments_2.expt:      "mask": "/dls/science/users/hko55533/mx21314-30/VMXi-AB0813/well_151_image_17607/DEFAULT/NATIVE/SWEEP1/index/2_hot_mask_0.pickle", 
vmxi_proteinase_k_sweeps/README.md:files="/dls/mx/data/mx21314/mx21314-30/VMXi-AB0813/well_12/images/image_17586_master.h5
vmxi_proteinase_k_sweeps/README.md:/dls/mx/data/mx21314/mx21314-30/VMXi-AB0813/well_46/images/image_17657_master.h5
vmxi_proteinase_k_sweeps/README.md:/dls/mx/data/mx21314/mx21314-30/VMXi-AB0813/well_112/images/image_17578_master.h5
vmxi_proteinase_k_sweeps/README.md:/dls/mx/data/mx21314/mx21314-30/VMXi-AB0813/well_151/images/image_17607_master.h5"
mpro-x0305_processed/indexed.expt:      "template": "/dls/i04-1/data/2020/lb18145-154/Mpro/Mpro-x0305/Mpro-x0305_1_####.cbf",
mpro-x0305_processed/imported.expt:      "template": "/dls/i04-1/data/2020/lb18145-154/Mpro/Mpro-x0305/Mpro-x0305_1_####.cbf",
mpro-x0305_processed/README.md:dials.import /dls/i04-1/data/2020/lb18145-154/Mpro/Mpro-x0305/*.cbf
pdteet-2-lattice/sweep-04/indexed.expt:      "template": "/Users/graeme/data/i19-1-double-PdTEET/PdTEET_04_#####.cbf.gz", 
l_cysteine_4_sweeps_scaled/scaled_20_25.expt:      "template": "/Users/graeme/data/i19-1/l-cyst/l-cyst_04_#####.cbf", 
l_cysteine_4_sweeps_scaled/scaled_20_25.expt:      "mask": "/Users/graeme/Projects/l-cys/DEFAULT/NATIVE/SWEEP1/index/hot_mask_0.pickle", 
l_cysteine_4_sweeps_scaled/scaled_20_25.expt:      "template": "/Users/graeme/data/i19-1/l-cyst/l-cyst_01_#####.cbf", 
l_cysteine_4_sweeps_scaled/scaled_20_25.expt:      "mask": "/Users/graeme/Projects/l-cys/DEFAULT/NATIVE/SWEEP1/index/hot_mask_0.pickle", 
l_cysteine_4_sweeps_scaled/scaled_30.expt:      "template": "/Users/graeme/data/i19-1/l-cyst/l-cyst_02_#####.cbf", 
l_cysteine_4_sweeps_scaled/scaled_30.expt:      "mask": "/Users/graeme/Projects/l-cys/DEFAULT/NATIVE/SWEEP1/index/hot_mask_0.pickle", 
l_cysteine_4_sweeps_scaled/scaled_35.expt:      "template": "/Users/graeme/data/i19-1/l-cyst/l-cyst_03_#####.cbf", 
l_cysteine_4_sweeps_scaled/scaled_35.expt:      "mask": "/Users/graeme/Projects/l-cys/DEFAULT/NATIVE/SWEEP1/index/hot_mask_0.pickle", 
Binary file centroid_test_data/centroid_0007.cbf matches
Binary file centroid_test_data/centroid_0008.cbf matches
centroid_test_data/INTEGRATE.HKL:!NAME_TEMPLATE_OF_DATA_FRAMES=/Users/graeme/data/RU/1/X4_lots_M1S4_1_0???.cbf CB
Binary file centroid_test_data/centroid_0005.cbf matches
Binary file centroid_test_data/centroid_0003.cbf matches
Binary file centroid_test_data/centroid_0002.cbf matches
Binary file centroid_test_data/centroid_0006.cbf matches
Binary file centroid_test_data/centroid_0001.cbf matches
centroid_test_data/XDS.INP:!NAME_TEMPLATE_OF_DATA_FRAMES=/dls/i02/data/2013/nt5964-1/2013_02_08/GW/DNA/P1/X4/X4_lots_M1S4_1_????.cbf
Binary file centroid_test_data/centroid_0009.cbf matches
Binary file centroid_test_data/centroid_0004.cbf matches

@benjaminhwilliams benjaminhwilliams mentioned this issue Oct 12, 2020
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@benjaminhwilliams