Error while converting suite2p segmented metadata

[bendichter]

Describe the issue

Steps to Reproduce

Add suite2p data and try to convert

Operating System

OSX

GUIDE Version

latest release

Code of Conduct

Yes

Did you confirm this issue was not already reported?

Yes

[bendichter]

@CodyCBakerPhD this error happened while on a call with a PI who was trying to use NWB GUIDE. She downloaded the mac silicon version of the app today. Could you look into this and see if you could reproduce the error?

[bendichter]

I tried to reproduce and haven't been able to yet

[CodyCBakerPhD]

Do you remember anything about the suite2p setup? Was it multichannel, multiplane or something else more exotic?

[CodyCBakerPhD]

If you can get your hands on the data that caused it maybe it was something particular to that

[bendichter]

The data was:

suite2p/plane0/[.mat files, etc]

Only the one plane. The user selected the suite2p folder. As you can see, this was for a multi-session conversion.

I can try to get my hands on the suite2p data.

[bendichter]

I was able to reproduce it.

First, I created a multi-session test directory by duplicating the test suite2p data:

Then, I ran through the multi-session workflow. I reached the import error:

Traceback (most recent call last):
  File "apis/neuroconv.py", line 97, in post
  File "manageNeuroconv/manage_neuroconv.py", line 387, in get_metadata_schema
  File "manageNeuroconv/manage_neuroconv.py", line 358, in instantiate_custom_converter
  File "neuroconv/nwbconverter.py", line 70, in __init__
    self.data_interface_objects = {
  File "neuroconv/nwbconverter.py", line 71, in <dictcomp>
    name: data_interface(**source_data[name])
  File "neuroconv/datainterfaces/ophys/suite2p/suite2pdatainterface.py", line 85, in __init__
    super().__init__(folder_path=folder_path, channel_name=channel_name, plane_name=plane_name)
  File "neuroconv/datainterfaces/ophys/basesegmentationextractorinterface.py", line 23, in __init__
    self.segmentation_extractor = self.get_extractor()(**source_data)
  File "neuroconv/baseextractorinterface.py", line 26, in get_extractor
    extractor = getattr(
AttributeError: partially initialized module 'roiextractors' has no attribute 'Suite2pSegmentationExtractor' (most likely due to a circular import)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "flask/app.py", line 1484, in full_dispatch_request
  File "flask/app.py", line 1469, in dispatch_request
  File "flask_restx/api.py", line 404, in wrapper
  File "flask/views.py", line 109, in view
  File "flask_restx/resource.py", line 46, in dispatch_request
  File "apis/neuroconv.py", line 102, in post
  File "flask_restx/namespace.py", line 153, in abort
  File "flask_restx/errors.py", line 28, in abort
  File "flask/helpers.py", line 277, in abort
  File "werkzeug/exceptions.py", line 861, in __call__
werkzeug.exceptions.InternalServerError: 500 Internal Server Error: The server encountered an internal error and was unable to complete your request. Either the server is overloaded or there is an error in the application.

here's another error (which may be related)

[POST]
Traceback (most recent call last):
  File "apis/neuroconv.py", line 97, in post
  File "manageNeuroconv/manage_neuroconv.py", line 387, in get_metadata_schema
  File "manageNeuroconv/manage_neuroconv.py", line 358, in instantiate_custom_converter
  File "neuroconv/nwbconverter.py", line 70, in __init__
    self.data_interface_objects = {
  File "neuroconv/nwbconverter.py", line 71, in <dictcomp>
    name: data_interface(**source_data[name])
  File "neuroconv/datainterfaces/ophys/suite2p/suite2pdatainterface.py", line 85, in __init__
    super().__init__(folder_path=folder_path, channel_name=channel_name, plane_name=plane_name)
  File "neuroconv/datainterfaces/ophys/basesegmentationextractorinterface.py", line 23, in __init__
    self.segmentation_extractor = self.get_extractor()(**source_data)
  File "roiextractors/extractors/suite2p/suite2psegmentationextractor.py", line 155, in __init__
TypeError: 'NoneType' object is not subscriptable

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "flask/app.py", line 1484, in full_dispatch_request
  File "flask/app.py", line 1469, in dispatch_request
  File "flask_restx/api.py", line 404, in wrapper
  File "flask/views.py", line 109, in view
  File "flask_restx/resource.py", line 46, in dispatch_request
  File "apis/neuroconv.py", line 102, in post
  File "flask_restx/namespace.py", line 153, in abort
  File "flask_restx/errors.py", line 28, in abort
  File "flask/helpers.py", line 277, in abort
  File "werkzeug/exceptions.py", line 861, in __call__
werkzeug.exceptions.InternalServerError: 500 Internal Server Error: The server encountered an internal error and was unable to complete your request. Either the server is overloaded or there is an error in the application.

I got these errors by pressing Next on this screen:

[bendichter]

Interestingly, this does not appear to occur when working with a single session

[bendichter]

OK, so when I had all the files, including the ops.npy file, in the proper place, the conversion worked perfectly. When then ops.npy file is missing we get a TypeError: 'NoneType' object is not subscriptable error. This error is for some reason cut off in the developer console and I could only figure out once I ran this in dev mode and saw the error in the stdout. I opened an issue for this. If the ops.npy file is required, then we should ensure it is there. I have opened an issue for this: catalystneuro/roiextractors#329

[bendichter]

I think the circular import error may be a strange consequence of the TypeError: 'NoneType' object is not subscriptable error

[CodyCBakerPhD]

I think the solution to this particular error was to install from latest main branch of ROI Extractors?

[bendichter]

I am not entirely sure, but I think there were a few issues here.

The first one is that ROIExtractors was not enforcing the presence of files that should be required: catalystneuro/roiextractors#330

The second is that this error somehow caused there to be an import error which is completely separate from the core problem

The third was that the user did not have proper access to the input data, which is the reason this was triggered in the first place.

I think this has been resolved. If this comes up again let's reopen.