Replies: 9 comments 12 replies
-
|
Examples for types of results files that would need to be described with a generalized "data dictionary"
@muehlhaus @Brilator adding examples for TDF and MAF files |
Beta Was this translation helpful? Give feedback.
-
|
The first idea that came up in DataPLANT is to extend the ISA “data file” object with a kind of data dictionary including the following fields: We envision an additional file (table) called “ISA dataset” that accompanies one ore more result files (e.g. data matrices). Your thoughts are highly appreciated. |
Beta Was this translation helpful? Give feedback.
-
|
Defining the structure for the
|
Beta Was this translation helpful? Give feedback.
-
|
Hey @proccaserra, I don't understand the |
Beta Was this translation helpful? Give feedback.
-
|
@Brilator the cardinality is meant specify how many times a field is expected to be found in a target file. But you are right in the sense that the string 'sample identifier' does not occur explicitly in the file itself. I was trying to capture that fact by using a combination of 'cardinality' and 'regex' but this is in not correct. So it seems another mechanism is needed to indicate fields which 1/ follow a pattern and 2/ can occur multiple times . Typically, an array of sample/bioassay identifiers or an array of contrasts |
Beta Was this translation helpful? Give feedback.
-
|
@muehlhaus ,is this what you had in mind (please see the second example where 2 files are referenced ( the MIAPPA TDF and the Metabolights MAF). this brings up the issue of complex header structure to take into account 2 dimensions: |
Beta Was this translation helpful? Give feedback.
-
|
Hi @proccaserra, exactly this are my intentions. The `data dictionary defines each of the fields (headers). 'Data cube orientation' is a valid concern. A possible solution would be to include the orientation as an additional field in the dictionary (row/column major) or we require column major by definition (maybe less favorable). Regarding the This would also make the description of multiple dimensions possible... => e.g. same identifier different source |
Beta Was this translation helpful? Give feedback.
-
|
@muehlhaus I now got what you meant by 'Identifier'. I got wrong-footed and understood it as a Note that if users were to use for example Frictionless.io, the data files referenced from an ISA document would be somehow self describing. Let's continue this discussion and provide more examples. thank you all! |
Beta Was this translation helpful? Give feedback.
-
|
@proccaserra I think that Let`s assume in column 1 and 2 of our data matrix are quantification values (of any kind), I want to know that column 1 was obtained under 25°C while column 2 was obtained under 45°C ... This is information that we have stored in the ISA Graph. However, currently the connection is not in the model. Using frictionless.io to solve this, would require a link/definition from frictionless.io to ISA... and just postpone the issue (at least to my understanding). Also, it would require loading information from ISA and frictionless description... |
Beta Was this translation helpful? Give feedback.
-
|
Hello. My (computer scientist) pov. The following is more or less how I (and a few others) have envisioned the "data dictionary" (not necessarily complying with ISA, actually kind of orthogonal to it).
The following (imagined as an additional sheet in "isa.dataset.xlsx") is less of an extension but more a reuse of ISA to document which physical samples contributed (via some software) to which data files. (The column names are rather ad-hoc, excuse me, I am not much of an ISA expert yet.)
Lastly, the following pic demonstrates (roughly …) my intended use case: being able to grasp individual "variables" inside data files.
|
Beta Was this translation helpful? Give feedback.
-
|
Oh, to clarify that empty column. There is the concept of "derived" variables, that – in my mock-up – go in different sheets depending on their "level of dependence". E.g.
and L2-Derivations
|
Beta Was this translation helpful? Give feedback.
-
|
Nice work. Let me look it up.
…On Mon, Jan 9, 2023, 17:56 Christopher Kappe ***@***.***> wrote:
Oh, to clarify that empty column. There is the concept of "derived"
variables, that – in my mock-up – go in different sheets depending on their
"level of dependence". E.g.
L1-Derivations
Data File Path Source Name Software Data File Path 2 Sample Name
runs/kallisto_sleuth/sleuth_dge.csv mean_obs sleuth
runs/kallisto_sleuth/sleuth_dge.csv smooth_sigma_sq
runs/kallisto_sleuth/sleuth_dge.csv sigma_sq sleuth
runs/kallisto_sleuth/sleuth_dge.csv smooth_sigma_sq
and L2-Derivations
Data File Path Source Name Software Data File Path 2 Sample Name
runs/kallisto_sleuth/sleuth_dge.csv sigma_sq sleuth
runs/kallisto_sleuth/sleuth_dge.csv rss
runs/kallisto_sleuth/sleuth_dge.csv smooth_sigma_sq sleuth
runs/kallisto_sleuth/sleuth_dge.csv rss
runs/kallisto_sleuth/sleuth_dge.csv sigma_sq sleuth
runs/kallisto_sleuth/sleuth_dge.csv final_sigma_sq
runs/kallisto_sleuth/sleuth_dge.csv smooth_sigma_sq sleuth
runs/kallisto_sleuth/sleuth_dge.csv final_sigma_sq
—
Reply to this email directly, view it on GitHub
<#484 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/A3K3NVVPCC3QWGCXOPKEXY3WRQ7MZANCNFSM6AAAAAAS6WHRJQ>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
Beta Was this translation helpful? Give feedback.
-
|
Hey, I also wanted to give my input from the perspective of a tool developer, consuming both Process graph pointersFirstly I want to touch on the pointers in the process graph, which can then be used to associate Naming of the headers / Integration into ISA-TabAs was stated above, these So these header pairs (
Having Integration into ISA-JsonThis is relatively straight forward I think. The data object currently contains the field {
"filePath": "runs/kallisto_sleuth/sleuth_dge.csv",
"identifier": "mean_obs",
"type": "Raw Data File"
}and {
"filePath": "runs/kallisto_sleuth/sleuth_dge.csv",
"identifier": "smooth_sigma_sq",
"type": "Derived Data File"
}The name "type": {
"type": "string",
"enum": [
"Raw Data File",
"Derived Data File",
"Image File",
"Raw Data",
"Derived Data",
"Image"
]
}or just "type": {
"type": "string",
"enum": [
"Raw Data",
"Derived Data",
"Image"
]
}Pointer descriptors / DatasetAll of you touched the reasons for and the modelling details of these pointer descriptors nicely. I just want to add a few thoughts about the implementation. Integration into ISA-TabI'm not quite sure what your final thoughts about the STUDY
Unfortunately this breaks the DATASET
Input on this would be much appreciated. Integration into ISA-JsonAgain, the integration into {
"filePath": "runs/kallisto_sleuth/sleuth_dge.csv",
"pointer": "mean_obs",
"type": "Raw Data File",
"wasGeneratedBy" : "workflows/kallisto_sleuth.R",
"attribute" : {
"annotationValue" : "Arithmetic Mean",
"termSource" : "NCIT",
"termAccession" : "http://purl.obolibrary.org/obo/NCIT_C53319"
},
"objectType" : "Decimal",
"label" : "Mean"
} |
Beta Was this translation helpful? Give feedback.
-
|
@HLWeil @kappe-c, thank you the input and explanation. @HLWeil : one clarification please. Should your I am still unclear about the following points:
@terazus what are your thoughts ? |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for your answer, @proccaserra. Regarding your other points:
--
Explanation for Dataset/Data dictionary: Initially, the |
Beta Was this translation helpful? Give feedback.
-
|
Hi guys, I just wanted to follow up on this. |
Beta Was this translation helpful? Give feedback.
-
|
hi @kappe-c @muehlhaus @HLWeil , We've been giving more thoughts about the proposal and we came to the realisation that the specification outlined is outside the scope of ISA. It is complementary though. We all understand and agree on the need to describe computational process and results. I don't think we have a generic solution yet. I agree with @HLWeil about another solution: i.e. add an entirely new section "DATA ANALYSIS" , which would cover the computation workflow executions, describe the input files, output files and their header definitions (if tabular), or other descriptions if graphical output, rules for combining fields. but this sounds very much like a new workflow language. |
Beta Was this translation helpful? Give feedback.
-
|
Thank you for the info @proccaserra ! I guess we will then think about how to concretly advance this topic outside of ISA for now. |
Beta Was this translation helpful? Give feedback.
-
|
@proccaserra I agree that this seems out of the scope of ISA. If I understand correctly what is wanted I'm not sure it could realistically be achieved or would even be a good idea. The way I am understanding this thread is that people want a way to describe how the data is arranged inside the various data files. This would vary greatly depending on the type of file, so I don't really see how a generalized solution could be done except on a file type basis. For instance, you could indicate a format for the file, such as "tabular" and then have a system for describing tabular files, but that gets complicated quickly. Maybe adding some minimal attributes to data files such as "format" and "specification" could be helpful for programmatically identifying well defined file types. For internal use, the comments on data files could be used to implement some of this already. |
Beta Was this translation helpful? Give feedback.
-
|
Hey @ptth222, I agree that this is a difficult endeavour, but a necessary one. Without explicit selection of the file fragments, there is no way to automatically associate final computational results with the experimental factors they are related to. |
Beta Was this translation helpful? Give feedback.
-
|
@HLWeil What you just linked is more reasonable and is quite different from everything else above. I did not know that most of the selector stuff was already worked out. Note that how the selector stuff works is exactly what I was talking about and is done on a file type basis. I have to disagree that file fragments are the only way to do this though. The other repository formats I have worked with create their own data structures that you have to conform yours to so it ends up directly linked. For instance, if ISA created a new measurement node and you had to extract all the data from the files into these nodes you get the associations without file fragments. Again, this is what some other places do (Metabolomics Workbench for example). Specifically, what you have done with the Data Format and Data Selector Format look very good.
Trying to work out how this would translate to ISA-JSON is interesting. Currently, the JSON doesn't associate entities (source, sample, material) with data files directly, and the selector stuff is a property of the linked entity, not the data file. I think the format needs to go on the data file. Connecting the entities to the file directly is more difficult. I'm thinking maybe a "measurements" list and the objects in the list would have 2 possible forms, either something like a characteristic (to handle cases of a single measurement), or a data file with additional "selector" and "selector format" properties. I think the "selector" has to be a property on its own since data files are given unique ids in the JSON and references within the JSON use those ids. "Selector" might have to be a separate column too to save parsing the file name value for it. Something like: This is a little bit different from some of the norms of the ISA-JSON format, but it solves 3 different problems:
The only other issue I see with file fragments is it introduces a new source of error. You can accidently indicate the wrong row for an entity for example. Assuming the file also had a way to link the entities, such as a key column, then it can be unclear whether the indicated row is wrong or the key column is wrong. I wouldn't say this is enough to scrap the idea though. |
Beta Was this translation helpful? Give feedback.
-
The community requires a way to describe result file content as part of the ISA -Model. Here, we want to discuss possible solutions. (Discussion related to issue #475)
Currently, the ISA model is very strong in describing the path from biological source to a measurement result file. From this point on the model relies on the specification of the result file format for machine tractability, which is perfect if such a file format is established. However, we often face the situation that such a file format is not established, or we want to point into such a format describing a specific processing path.
• Possibility to point into a file (e.g. data frame, XML, JSON, or image coordinate space)
• Using the classical ISA process description (e.g. ISA Tab) to not change the user experience
Beta Was this translation helpful? Give feedback.
All reactions