Create shelves files for pds4 #57

juzen2003 · 2024-09-23T22:53:23Z

Current status of creating shevles files for pds4:

Create files in checksums-* directory (pds4checksums.py)
- Modification made:
  - update BUNDLENAME_REGEX
- Example command:
  - python holdings_maintenance/pds4/pds4checksums.py --init /Volumes/rms-holdings/pds4-holdings/bundles/uranus_occs_earthbased
  - python holdings_maintenance/pds4/pds4checksums.py --init /Volumes/rms-holdings/pds4-holdings/metadata/uranus_occs_earthbased
  - python holdings_maintenance/pds4/pds4checksums.py --init /Volumes/rms-holdings/pds4-holdings/diagrams/uranus_occs_earthbased
- Pending items:
  - Wait for the finalized directory structure for Cassini
Create files in _infoshelf-* directory (pds4infoshelf.py), corresponding checksums files from the above steps are required
- Modification made:
  - properly import pds4checksums
- Example command:
  - python holdings_maintenance/pds4/pds4infoshelf.py --init /Volumes/rms-holdings/pds4-holdings/bundles/uranus_occs_earthbased
  - python holdings_maintenance/pds4/pds4infoshelf.py --init /Volumes/rms-holdings/pds4-holdings/metadata/uranus_occs_earthbased
  - python holdings_maintenance/pds4/pds4infoshelf.py --init /Volumes/rms-holdings/pds4-holdings/diagrams/uranus_occs_earthbased
Create files in _indexshelf-metadata (pds4indexshelf.py)
- Modification made:
  - Put BUNDLENAME_REGEX to Pds3File & Pds4File classes since they are different for pds3 & pds4
  - Add IDX_EXT and LBL_EXT to Pds3File & Pds4File to replace '.tab' & '.lbl' in pdsfile.py
    - pds4 label extension is .xml and idx extension is .csv
    - pds3 label extension is .lbl and idx extension is .tab
- Pending items:
  - Wait for label files (.xml) in metadata
Create files in _linkshelf-* directory (pds4linkshelf.py)
- Modification made:
  - remove .TXT in EXTS_WO_LABELS, .TXT could have a label in pds4
  - Add the intelligence to link a file to its correspsonding label if the file is in that label's file_name tags.
  - Add the intelligence to identify files like errata.txt, or checksum files that don't exist in the label nor exist in the csv. They are not part of the archive, so they don't have labels.
- Example command:
  - python holdings_maintenance/pds4/pds4linkshelf.py --init /Volumes/rms-holdings/pds4-holdings/bundles/uranus_occs_earthbased
- Pending items:
  - To crease linkshelf-metadata, wait for label files (.xml) in metadata
  - Wait for the finalized directory structure for Cassini

Pending items:

Work on rules for pds4 archive (work in progress)
Once metadata labels are added, update pds4indexshelf.py to create _indexshelf-metadata for pds4
Once metadata labels are added, update pds4linkshelf.py to create _linkshelf-metadata for pds4
Test pds4 with use_shelves_only set to True

Note:

These directories are generated using full holdings and uploaded to Dropbox
We don't bypass any directories now, ring_models and _support are included when running the scripts.

Shared-OPUS/pdsdata/pds4-holdings/checksums-bundles/cassini_iss
Shared-OPUS/pdsdata/pds4-holdings/checksums-bundles/cassini_uvis_solarocc_beckerjarmak2023
Shared-OPUS/pdsdata/pds4-holdings/checksums-bundles/cassini_vims
Shared-OPUS/pdsdata/pds4-holdings/checksums-bundles/uranus_occs_earthbased
Shared-OPUS/pdsdata/pds4-holdings/checksums-diagrams/uranus_occs_earthbased
Shared-OPUS/pdsdata/pds4-holdings/checksums-metadata/uranus_occs_earthbased
Shared-OPUS/pdsdata/pds4-holdings/_infoshelf-bundles/cassini_iss
Shared-OPUS/pdsdata/pds4-holdings/_infoshelf-bundles/cassini_uvis_solarocc_beckerjarmak2023
Shared-OPUS/pdsdata/pds4-holdings/_infoshelf-bundles/cassini_vims
Shared-OPUS/pdsdata/pds4-holdings/_infoshelf-bundles/uranus_occs_earthbased
Shared-OPUS/pdsdata/pds4-holdings/_infoshelf-diagrams/uranus_occs_earthbased
Shared-OPUS/pdsdata/pds4-holdings/_infoshelf-metadata/uranus_occs_earthbased
Shared-OPUS/pdsdata/pds4-holdings/_linkshelf-bundles/cassini_iss
Shared-OPUS/pdsdata/pds4-holdings/_linkshelf-bundles/cassini_uvis_solarocc_beckerjarmak2023
Shared-OPUS/pdsdata/pds4-holdings/_linkshelf-bundles/cassini_vims
Shared-OPUS/pdsdata/pds4-holdings/_linkshelf-bundles/uranus_occs_earthbased

absolute path of a filespec.

…for_pds4

checksums-bundles directory for pds4

…for_pds4

_infoshelf-bundles

…for_pds4

_linkshelf-* directory

rfrenchseti · 2024-09-23T22:57:25Z

The --volume argument really is supposed to be just volume so that you put the positional argument on the command line without specifying any flag in front of it.

rfrenchseti · 2024-09-23T23:01:14Z

For metadata, you're just talking about the index files in uranus_occs_earthbased, right? Those are temporary files until Emilie's PDS4 index file generator is finished, at which point we will replace them with new files that also include the XML labels. There should never be files in the PDS4 archive that don't have associated labels.

rfrenchseti · 2024-09-23T23:04:31Z

I don't see any reason to skip ring_models. We should generate link files for all directories. The REPAIRS and KNOWN_MISSING_LABELS variables define places where the PDS3 labels were badly written and refer to the wrong files, directories, have spelling errors, etc. We hope that this will basically never happen in PDS4 labels. In any case, these variables are PDS3-volume-specific, and should be empty when starting from scratch with PDS4 until we see a reason to put something in them.

rfrenchseti · 2024-09-23T23:07:24Z

Before you get too far into pds4archives, we need to talk about how that's going to work. The PDS3 concept of archiving each volume separately does not work with PDS4 bundles, because, for example, the entire Cassini ISS Saturn mission is a single bundle (currently 116 PDS3 volumes), which would result in an enormous and impossible-to-use single archive file around a terabyte. This would be a good thing to discuss with the group at large. Of course how we split things into archives will also affect how we make checksum files for those archives.

rfrenchseti · 2024-09-24T19:00:47Z

Thinking more about this, we also can't ignore the _support directories. Remember that PdsFile has two purposes - OPUS and Viewmaster. For OPUS, we ignore all sorts of things because they aren't data products. But Viewmaster has to be able to handle every single file in the holdings/pds4-holdings directories, no matter what. Random documentation, text files, labels, data files, whatever shows up. Because the user will be browsing the entire holdings while using Viewmaster, and every single file needs to show up in Viewmaster with size, data, type, and maybe some intelligent commentary. So everything has to be in the shelve files. Everything has to be in the archives. Everything has to be in the checksums. Everything (relevant) has to be in the links. And, equally important, we need lots of rules in Pds4File for the PDS4 bundles that we don't currently have because OPUS doesn't need them - associations between file types, etc.

checksums, infoshelf, and linkshelf files.

file is in that label's file_name tags. (line 323-336, pds4linkshelf.py)

directory (line 149-154, pds4linkshelf.py)

files that don't exist in the label nor exist in the csv. They are not part of the archive, so they don't have labels. (line 369-378 in pds4linkshelf.py)

juzen2003 · 2024-10-11T20:43:59Z

Update the latest status on the top comments (10/12/24)

Current pending items:

Fix volset_ and volname_ backward compatibility for pds3 (work in progress)
Work on rules for pds4 archive (work in progress)
Once Cassini directory structured is finalized, update and run the scripts.
Once metadata labels are added, update pds4indexshelf.py to create _indexshelf-metadata for pds4
Once metadata labels are added, update pds4linkshelf.py to create _linkshelf-metadata for pds4
Test pds4 with use_shelves_only set to True

juzen2003 · 2024-10-11T20:45:58Z

The --volume argument really is supposed to be just volume so that you put the positional argument on the command line without specifying any flag in front of it.

Fixed

juzen2003 · 2024-10-11T20:46:15Z

I don't see any reason to skip ring_models. We should generate link files for all directories. The REPAIRS and KNOWN_MISSING_LABELS variables define places where the PDS3 labels were badly written and refer to the wrong files, directories, have spelling errors, etc. We hope that this will basically never happen in PDS4 labels. In any case, these variables are PDS3-volume-specific, and should be empty when starting from scratch with PDS4 until we see a reason to put something in them.

Fixed

juzen2003 · 2024-10-11T20:46:28Z

Thinking more about this, we also can't ignore the _support directories. Remember that PdsFile has two purposes - OPUS and Viewmaster. For OPUS, we ignore all sorts of things because they aren't data products. But Viewmaster has to be able to handle every single file in the holdings/pds4-holdings directories, no matter what. Random documentation, text files, labels, data files, whatever shows up. Because the user will be browsing the entire holdings while using Viewmaster, and every single file needs to show up in Viewmaster with size, data, type, and maybe some intelligent commentary. So everything has to be in the shelve files. Everything has to be in the archives. Everything has to be in the checksums. Everything (relevant) has to be in the links. And, equally important, we need lots of rules in Pds4File for the PDS4 bundles that we don't currently have because OPUS doesn't need them - associations between file types, etc.

Fixed

subclasses.

…for_pds4

in the file_name tags and avoid capturing the file name in the title tag of the label. This will prevent us from getting duplicated file name of the LinkInfo object when the file name exists in the title tag.

pds4linkshelf.py.

- moving the intelligence to check if a file is in the file_name tag of a label. Now this step is done after checking whether the file is in the label_dict already. - moving the intelligence to check if a file is in the collection csv files. Now this step is done right before raising an error when we can't find its corresponding label. These two modifications can avoid unnecessary looping of linkinfo_dict and collection_basename_dict.

when trying to parse each entry to get the basename of a file in the archive.

juzen2003 · 2024-10-21T19:39:45Z

Update the latest status, the top comments are also updated (10/22/24)

Update maintenance tools under holdings_maintenance/pds4 to generate checksums, infoshelf, and linkshelf for PDS4 bundles
These Newly generated shelf files are uploaded to Dropbox

Shared-OPUS/pdsdata/pds4-holdings/checksums-bundles/cassini_iss
Shared-OPUS/pdsdata/pds4-holdings/checksums-bundles/cassini_uvis_solarocc_beckerjarmak2023
Shared-OPUS/pdsdata/pds4-holdings/checksums-bundles/cassini_vims
Shared-OPUS/pdsdata/pds4-holdings/checksums-bundles/uranus_occs_earthbased
Shared-OPUS/pdsdata/pds4-holdings/checksums-diagrams/uranus_occs_earthbased
Shared-OPUS/pdsdata/pds4-holdings/checksums-metadata/uranus_occs_earthbased
Shared-OPUS/pdsdata/pds4-holdings/_infoshelf-bundles/cassini_iss
Shared-OPUS/pdsdata/pds4-holdings/_infoshelf-bundles/cassini_uvis_solarocc_beckerjarmak2023
Shared-OPUS/pdsdata/pds4-holdings/_infoshelf-bundles/cassini_vims
Shared-OPUS/pdsdata/pds4-holdings/_infoshelf-bundles/uranus_occs_earthbased
Shared-OPUS/pdsdata/pds4-holdings/_infoshelf-diagrams/uranus_occs_earthbased
Shared-OPUS/pdsdata/pds4-holdings/_infoshelf-metadata/uranus_occs_earthbased
Shared-OPUS/pdsdata/pds4-holdings/_linkshelf-bundles/cassini_iss
Shared-OPUS/pdsdata/pds4-holdings/_linkshelf-bundles/cassini_uvis_solarocc_beckerjarmak2023
Shared-OPUS/pdsdata/pds4-holdings/_linkshelf-bundles/cassini_vims
Shared-OPUS/pdsdata/pds4-holdings/_linkshelf-bundles/uranus_occs_earthbased

juzen2003 added 28 commits December 6, 2023 13:25

Add validation directory (moved from rms-webtools repo)

2e94c7b

Merge remote-tracking branch 'origin/main'

4be35d4

Merge remote-tracking branch 'origin/main'

5e5fe97

Merge remote-tracking branch 'origin/main'

fd70eb6

Merge remote-tracking branch 'origin/main'

9f9c733

Merge remote-tracking branch 'origin/main'

077c359

Merge remote-tracking branch 'origin/main'

291555e

Merge remote-tracking branch 'origin/main'

d195a79

Merge remote-tracking branch 'origin/main'

1f762cb

Merge remote-tracking branch 'origin/main'

92bd65c

Merge remote-tracking branch 'origin/main'

205b9df

Add command line tool to show opus products output with the given

ab669f9

absolute path of a filespec.

Merge remote-tracking branch 'origin/main'

c7d84e5

Merge remote-tracking branch 'origin/main'

a075bbc

Merge remote-tracking branch 'origin/main'

428d727

Merge remote-tracking branch 'origin/main' into create_shelves_files_…

fd4c7b6

…for_pds4

Merge remote-tracking branch 'origin/main'

756547c

Merge remote-tracking branch 'origin/main'

d2d3803

Merge branch 'main' into create_shelves_files_for_pds4

a875899

Create pds4 directory to store maintenance files for pds4.

4ffbc45

Merge remote-tracking branch 'origin/main' into create_shelves_files_…

4b9b154

…for_pds4

Update pds4/pds4checksums.py so that checksum files can be created under

cd66002

checksums-bundles directory for pds4

Merge remote-tracking branch 'origin/main' into create_shelves_files_…

fe7c687

…for_pds4

Update pds4/pds4infoshelf.py to create info sheleves files under

ddbbb2a

_infoshelf-bundles

Removed the debug print statement in the pdsfile.py

69bfd7a

Merge remote-tracking branch 'origin/main' into create_shelves_files_…

cb08184

…for_pds4

Create pds4/pds4indexshelf.py & pds4/pds4linkshelf.py

bc74047

Update pds4linkshelf.py to make sure we can create files in

5509b5a

_linkshelf-* directory

juzen2003 marked this pull request as draft September 23, 2024 22:53

juzen2003 added 5 commits September 25, 2024 10:00

Remove the code that bypass _support & ring_modeles when creating

b0a7944

checksums, infoshelf, and linkshelf files.

Add an intelligence to link a file to its correspsonding label if the

10b6743

file is in that label's file_name tags. (line 323-336, pds4linkshelf.py)

Fixed the code indentation from line 112-428 in pds4linkshelf.py

c3f1572

Add the intelligence to ignore files when there is no label at the same

0ef4578

directory (line 149-154, pds4linkshelf.py)

Add the intelligence to identify files like errata.txt, or checksum

3f3d2da

files that don't exist in the label nor exist in the csv. They are not part of the archive, so they don't have labels. (line 369-378 in pds4linkshelf.py)

juzen2003 added 9 commits October 16, 2024 05:00

Separate BUNDLENAME* and BUNDLESET* regex, put them under PDS3 & PDS4

a7c725d

subclasses.

Merge remote-tracking branch 'origin/main' into create_shelves_files_…

1890a8f

…for_pds4

Removed redundant BUNDLESET* and BUNDLENAME* regex for PDS3.

e059710

Change is_volume_dir to is_bundle_dir in pds4checksums.py

d83b146

Update read_links function in pds4linkshelf.py to capture file names

1c06fe9

in the file_name tags and avoid capturing the file name in the title tag of the label. This will prevent us from getting duplicated file name of the LinkInfo object when the file name exists in the title tag.

Remove the commented out code in the read_links function of

0eea1fe

pds4linkshelf.py.

Remove commented out logger.debug generate_links

8fa331d

At line 156-158 of pds4linkshelf.py, skip the empty entry of a csv file

39639c6

when trying to parse each entry to get the basename of a file in the archive.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Create shelves files for pds4 #57

Create shelves files for pds4 #57

juzen2003 commented Sep 23, 2024 •

edited

Loading

rfrenchseti commented Sep 23, 2024

rfrenchseti commented Sep 23, 2024

rfrenchseti commented Sep 23, 2024

rfrenchseti commented Sep 23, 2024

rfrenchseti commented Sep 24, 2024

juzen2003 commented Oct 11, 2024

juzen2003 commented Oct 11, 2024

juzen2003 commented Oct 11, 2024

juzen2003 commented Oct 11, 2024

juzen2003 commented Oct 21, 2024

Create shelves files for pds4 #57

Are you sure you want to change the base?

Create shelves files for pds4 #57

Conversation

juzen2003 commented Sep 23, 2024 • edited Loading

rfrenchseti commented Sep 23, 2024

rfrenchseti commented Sep 23, 2024

rfrenchseti commented Sep 23, 2024

rfrenchseti commented Sep 23, 2024

rfrenchseti commented Sep 24, 2024

juzen2003 commented Oct 11, 2024

juzen2003 commented Oct 11, 2024

juzen2003 commented Oct 11, 2024

juzen2003 commented Oct 11, 2024

juzen2003 commented Oct 21, 2024

juzen2003 commented Sep 23, 2024 •

edited

Loading