Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

How to use the module config options #27

Open
MariellaCC opened this issue Feb 7, 2024 · 5 comments
Open

How to use the module config options #27

MariellaCC opened this issue Feb 7, 2024 · 5 comments

Comments

@MariellaCC
Copy link

Use case: I need to only onboard text files from a folder and I am using the import.local.file_bundle module.

When using the module without config options, some undesired DS_store file gets onboarded, but I am only interested in .txt files.

I did a kiara operation explain import.local.file_bundle -m in the terminal to see if config options are available for this module, and found "include_file_types" and "exclude_file_types" that seem available:

╭─ Operation: import.local.file_bundle ───────────────────────────────────────────────────────────────────────────────────────────────────╮
│                                                                                                                                         │
│   Documentation     Import a folder (file_bundle) from the local filesystem.                                                            │
│                                                                                                                                         │
│   Inputs                                                                                                                                │
│                       field name   type     description                                                 Required   Default              │
│                      ───────────────────────────────────────────────────────────────────────────────────────────────────────────────    │
│                       path         string   The local path of the folder to import.                     yes        -- no default --     │
│                                                                                                                                         │
│                                                                                                                                         │
│   Outputs                                                                                                                               │
│                       field name    type          description                                                                           │
│                      ───────────────────────────────────────────────────────────────────────────────────────────────────────────────    │
│                       file_bundle   file_bundle   The imported file bundle.                                                             │
│                                                                                                                                         │
│                                                                                                                                         │
│   Module type       import.local.file_bundle                                                                                            │
│                                                                                                                                         │
│   Module config     {                                                                                                                   │
│                       "constants": {},                                                                                                  │
│                       "defaults": {},                                                                                                   │
│                       "include_file_types": null,                                                                                       │
│                       "exclude_file_types": null                                                                                        │
│                     }                                                                                                                   │
│                                                                                                                                         │
│   Module metadata   Import a folder (file_bundle) from the local filesystem.                                                            │
│                                                                                                                                         │
│                      Author(s)                                                                                                          │
│                                     Markus Binsteiner   [email protected]                                                                  │
│                                                                                                                                         │
│                      Context                                                                                                            │
│                                     Labels       package: kiara                                                                         │
│                                     References   source_repo: https://github.com/DHARPA-Project/kiara                                   │
│                                                  documentation: https://dharpa.org/kiara_documentation/                                 │
│                                                                                                                                         │
│                      Python class                                                                                                       │
│                                     python_class_name    ImportLocalFileBundleModule                                                    │
│                                     python_module_name   kiara.modules.included_core_modules.filesystem                                 │
│                                     full_name            kiara.modules.included_core_modules.filesystem.ImportLocalFileBundleModu…      │
│                                                                                                                                         │
│                                                                                                                                         │
│                                                                                                                                         │
╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯

To learn more about what kind of inputs are expected here, I did kiara module explain import.local.file_bundle, which resulted in the following info:

 
╭─ Module type: import.local.file_bundle ─────────────────────────────────────────────────────────────────────────────────────────────────╮
│                                                                                                                                         │
│  Documentation                                                                                                                          │
│                         Import a folder (file_bundle) from the local filesystem.                                                        │
│                                                                                                                                         │
│  Author(s)                                                                                                                              │
│                         Markus Binsteiner   [email protected]                                                                              │
│                                                                                                                                         │
│  Context                                                                                                                                │
│                         Labels       package: kiara                                                                                     │
│                         References   source_repo: https://github.com/DHARPA-Project/kiara                                               │
│                                      documentation: https://dharpa.org/kiara_documentation/                                             │
│                                                                                                                                         │
│  Module config schema                                                                                                                   │
│                         Field                Type                 Description                        Required   Default                 │
│                        ─────────────────────────────────────────────────────────────────────────────────────────────────                │
│                         constants            object               Value constants for this module.   no                                 │
│                                                                                                                                         │
│                         defaults             object               Value defaults for this module.    no                                 │
│                                                                                                                                         │
│                         exclude_file_types   -- check source --   File types to include.             no                                 │
│                                                                                                                                         │
│                         include_file_types   -- check source --   File types to include.             no                                 │
│                                                                                                                                         │
│  Python class                                                                                                                           │
│                         python_class_name    ImportLocalFileBundleModule                                                                │
│                         python_module_name   kiara.modules.included_core_modules.filesystem                                             │
│                         full_name            kiara.modules.included_core_modules.filesystem.ImportLocalFileBundleModule                 │
│                                                                                                                                         │
│                                                                                                                                         │
╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯                                                                                                                    

The info provided "check source" to learn about how I should format the input and what inputs are accepted refers to the source repo info indicated: source_repo: https://github.com/DHARPA-Project/kiara.

@makkus my question is the following, how do I find easily the import.local.file_bundle within https://github.com/DHARPA-Project/kiara to be able to check what input I can use?

@makkus
Copy link
Collaborator

makkus commented Feb 7, 2024

So, in this case you'd find the file that contains the class 'kiara.modules.included_core_modules.filesystem.ImportLocalFileBundleModule' that is displayed in the Python class value (either via the github repo or using your IDEs discovery mechanism (depends on your IDE, I use Pycharm and it's fairly easy to get to that class there). Lets assume github:

https://github.com/DHARPA-Project/kiara/blob/6cca099ff5fe1846fc822c129e2ff27c3f66009b/src/kiara/modules/included_core_modules/filesystem.py#L121

Once you have the Module class, you check the classes _config_cls class attribute, which is 'ImportFileBundleConfig' in our case:

https://github.com/DHARPA-Project/kiara/blob/6cca099ff5fe1846fc822c129e2ff27c3f66009b/src/kiara/modules/included_core_modules/filesystem.py#L111

Here you can check the type-hints for the config options you are interested in. (Sidenote: just realized there is a typo there and one 'include' should read 'exclude' -- will fix that now)

Further reading would be pydantic ( https://docs.pydantic.dev/latest/ ), because all Module config classes inherit from the pydantic BaseModel class. But for your purpose that shouldn't be necessary.

It's not trivial to surface that info through to a user interface, and so far I did not want to spend the time trying to do that because I imagine this to be an advanced use-case, that either wouldn't happen too often, or it would happen to people who are comfortable navigating source code this way. I can try to implement it though, if there is demand, as long as everybody thinks its important enough.

@MariellaCC
Copy link
Author

Thanks, @makkus, this is very helpful.

I would now like to check whether the string needs to include the point of the extension (e.g. ".txt" or "txt").
I guess that my next step would be to have a look at how a "KiaraFileBundle" works?
How could I find where the "KiaraFileBundle" is located? Is it a Kiara data type or something else?

@makkus
Copy link
Collaborator

makkus commented Feb 7, 2024

Right, so, this is something that I probably should have put into the description of those options, which I'll do now. Come to think of it, I can probably also include a note that those options are list of strings...

Anyway, still a good idea to know how to figure that one out, so...

You can find the 'KiaraFileBundle' class by looking at where it is imported:

https://github.com/DHARPA-Project/kiara/blob/6cca099ff5fe1846fc822c129e2ff27c3f66009b/src/kiara/modules/included_core_modules/filesystem.py#L16

Which would lead you to:

https://github.com/DHARPA-Project/kiara/blob/6cca099ff5fe1846fc822c129e2ff27c3f66009b/src/kiara/models/filesystem.py#L234

and the 'import_folder' method:

https://github.com/DHARPA-Project/kiara/blob/6cca099ff5fe1846fc822c129e2ff27c3f66009b/src/kiara/models/filesystem.py#L298

Which should show you after a bit of code-reading that the 'ends_with' test is used for both of those, so as long as your filename ends with one of the included/excluded strings, there's a match. Whether you include or exclude a dot doesn't matter.

Jumping around in the source code like this is essential to efficient programming, in my view, so I'd recommend figuring out how your IDE makes that easy for you. In Pycharm which I use, I can just do Shift-Shift and the name of the class, and it leads me there, or right-clicking on the class name and 'goto declaration' also does the same thing.
I'd be surprised if there wasn't something in VSCode that does the same thing, so if you haven't already, try to find that option (maybe it's called 'goto definition' instead, something along those lines...).

@MariellaCC
Copy link
Author

thanks,
just a detail but I could find out quite quickly that it was a list of strings that is needed, but wanted to figure out if there is a way the strings need to be (with or without the point "txt" or ".txt" inside a given string).

@makkus
Copy link
Collaborator

makkus commented Feb 7, 2024

Yeah, fair enough. Like I said, I should have documented that in the description of the options that the test is against the end of the file path, and as long as that matches you're good, with or without the point. It should not be necessary to jump through the source code to find that...

Still, good to know how to efficiently jump through dependency source code :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants