Replies: 1 comment
-
@Galileo-Galilei mentioned this idea in the context of a user problem with MLflow https://kedro.hall.community/kedro-mlflow-netcdf-dataset-path-issue-dLxZxJzh1dsl |
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
Description
To get the correct (versioned) file path for a dataset is quite hard. There doesn't seem to be one way that works for both non-versioned and versioned datasets and/or local and remote datasets.
The main question here is why users need to access the file path. Not all datasets have a file path, e.g.
APIDataset
and so it's important to understand the true user need, before diving into solution.Context
dataset._filepath()
for non-versioned, local datasetdataset_get_load_path()
for versioned, local datasetsget_filepath_str(self._get_load_path(), self._protocol)
This might not even be the full list of ways to get the file path.
Important
This idea is based on observations from several Kedro engineers see e.g. #1778. However, we need a clear view on what user needs are when it comes to why they need the file path and what their use cases are. Any implementation should be preceded by user research: #1978
Beta Was this translation helpful? Give feedback.
All reactions