Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Define API for reader and image_mapper #143

Draft
wants to merge 73 commits into
base: master
Choose a base branch
from
Draft

Conversation

TjarkMiener
Copy link
Member

This PR suggests APIs for the reader and image_mapper. The configuration system and component scheme of ctapipe is adopted. Besides we adopt astropy tables for the batch generation. Reading is working in monoscopic and stereoscopic mode for DL1 images and R1 waveforms. The code properly process data with different array and telescope (divergent) pointings.

A subclass designed for the adv. trigger system processing R0 waveforms will be added in a separated PR.

–––
Closes #31 #104

For looping over a given dl1 table and a single dl1 event (charges, peak times and mask), we can now retrieve the 2D images (input of the CNNs) without init and running the dl1dh reader.
also create separate function for the trigger patches on R0 data
remove also apply IM functions because it can be now replace by the internal .map_image() function
If prefix camera_frame is in the file, the user should add this prefix to the config file.
added batch generator

removed dl1dh transform

dl1dh provide now a static batch and we get the relevant information about the labels in the data loader of ctlearn
now stored in dl0 monitoring tree
last dimension of sample was missing
replaced by new design
Mainly astropy table operations are now used to retrieve the exmaple identifiers for the stereo reading mode. Code base is therefore heavily reduced and operations are more efficient.

Moved parameter settings outside the dl1dh. User can request to also the read dl1b  parameters by passing a list of column names in the batch_generation()

Removed init skip when pandas hdf5 with example identifiers is provided. It is not needed anymore since we are now fast and efficient with astropy tables and their operations.

split transformation into sub-functions for better readability.
this is removing redundant code

astopy table operations should be used a retrieved sum(), min or max etc.
Everything related to the selection of the subarray is done with ctapipe now

Whenever a new file is processed, it checks the consistency of the SubarrayDescription to the reference which is the first provided file; this ensures that all files have the subarray.
It defines a reading API with two childs for reading in mono and stereo mode. Then, the Image and Waveform childs inherits from both child classes (mono and stereo). Finally, the trigger child only inherits from the mono child
this is somehow needed because we modify the batch Table for the Trigger subclass

fix trigger subclass
RealLSTCam is the cam geometry from LST in the lstchain data format
if obs id and event id is the same for different types, the code failed.
…ch should be a table

use iterrows for iteration over the batch

store also subarray trigger table as class attribute

make get_tel_pointing also external
while processing real stereo data we do not have the true shower id in the batch
SST1M pipe writes real data in data format v5.0.0. We can support this version for real data only.
SST1M write the event-wise pointing directly in dl1 monitoring
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

use more ctapipe functionality
3 participants