WebDataset Reader #230
WebDataset Reader #230
Conversation
|
@swetha097 - please resolve conflicts |
…097/rocAL into swbs/webdataset_PR_branch
| AsciiValues samples_ascii; | ||
| AsciiComponent ascii_component; | ||
| for (auto &elem : _map_content) { | ||
| std::cerr << "Name :\t " << elem.first; |
There was a problem hiding this comment.
In this function it is better to accumulate all prints in an std::string and issue one print statement at the end. Probably we can do it as another PR in all such instances
| if (_meta_data_reader) | ||
| THROW("A metadata reader has already been created") | ||
| if (_augmented_meta_data) | ||
| THROW("Metadata can only have a single output") | ||
|
|
||
| bool generate_index = (index_path[0] == '\0') ? true : false; | ||
| if (generate_index) | ||
| std::cerr << "Index file is not provided, it may take some time to infer it from the tar file"; |
There was a problem hiding this comment.
do you want to continue the rest of the code after this error? why are we not using ERRLOG macro
There was a problem hiding this comment.
This statement is printed if the user doesn't pass the idx file for the webdataset tar file. We have added support to infer the file contents from the tar file if its not present.
We cannot use ERR macro here as this is not an error but a warning that it will take some time for inferring file contents. There are INFO and WRN macros which can be used here but we are not enabling them in CMakelists. Only ERR and THROW macros are available for use
| int img_size = rocalGetImageNameLen(handle, image_name_length); | ||
| std::vector<char> img_name(img_size); | ||
| rocalGetImageName(handle, img_name.data()); | ||
| std::cerr << "\n Image name: " << img_name.data() << "\n \n"; |
There was a problem hiding this comment.
why are we using std::cerr here and cout in other places
There was a problem hiding this comment.
For the classification and detection pipelines, the image names and labels are printed using cerr while other metadata were printed using cout so we used cerr for the image name here. We'll change cout for all the prints for the webdataset pipeline
Contains support for