Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Uniform Conversion with Backend Configuration #821

Merged
merged 17 commits into from
Jun 4, 2024
Merged

Uniform Conversion with Backend Configuration #821

merged 17 commits into from
Jun 4, 2024

Conversation

garrettmflynn
Copy link
Member

This PR attempts to handle backend conversion using the run_conversion function from NeuroConv so that conversion progress bars (#778) can be viewed.

@CodyCBakerPhD For some reason, I'm unable to manually extract the default backend configuration object before passing to the run_conversion function. Can you take a look to see what's wrong? Here's the error that I get:

[2024-06-03 12:17:41,301] ERROR in app: Exception on /neuroconv/convert [POST]
concurrent.futures.process._RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/Users/garrettflynn/miniconda3/envs/nwb-guide/lib/python3.9/concurrent/futures/process.py", line 246, in _process_worker
    r = call_item.fn(*call_item.args, **call_item.kwargs)
  File "/Users/garrettflynn/Documents/GitHub/nwb-guide/src/pyflask/manageNeuroconv/manage_neuroconv.py", line 1161, in convert_to_nwb
    create_file(info, log_url=log_url)
  File "/Users/garrettflynn/Documents/GitHub/nwb-guide/src/pyflask/manageNeuroconv/manage_neuroconv.py", line 952, in create_file
    raise e
  File "/Users/garrettflynn/Documents/GitHub/nwb-guide/src/pyflask/manageNeuroconv/manage_neuroconv.py", line 931, in create_file
    converter.run_conversion(
  File "/Users/garrettflynn/Documents/GitHub/neuroconv/src/neuroconv/nwbconverter.py", line 229, in run_conversion
    configure_backend(nwbfile=nwbfile_out, backend_configuration=backend_configuration)
  File "/Users/garrettflynn/Documents/GitHub/neuroconv/src/neuroconv/tools/nwb_helpers/_configure_backend.py", line 41, in configure_backend
    neurodata_object = nwbfile_objects[object_id]
  File "/Users/garrettflynn/Documents/GitHub/hdmf/src/hdmf/utils.py", line 1066, in __getitem__
    return super().__getitem__(key)
KeyError: '1d3fed0c-8278-49db-b3e8-71b8c3cbf844'
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/garrettflynn/miniconda3/envs/nwb-guide/lib/python3.9/site-packages/flask/app.py", line 1484, in full_dispatch_request
    rv = self.dispatch_request()
  File "/Users/garrettflynn/miniconda3/envs/nwb-guide/lib/python3.9/site-packages/flask/app.py", line 1469, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
  File "/Users/garrettflynn/miniconda3/envs/nwb-guide/lib/python3.9/site-packages/flask_restx/api.py", line 404, in wrapper
    resp = resource(*args, **kwargs)
  File "/Users/garrettflynn/miniconda3/envs/nwb-guide/lib/python3.9/site-packages/flask/views.py", line 109, in view
    return current_app.ensure_sync(self.dispatch_request)(**kwargs)
  File "/Users/garrettflynn/miniconda3/envs/nwb-guide/lib/python3.9/site-packages/flask_restx/resource.py", line 46, in dispatch_request
    resp = meth(*args, **kwargs)
  File "/Users/garrettflynn/Documents/GitHub/nwb-guide/src/pyflask/namespaces/neuroconv.py", line 83, in post
    return convert_all_to_nwb(
  File "/Users/garrettflynn/Documents/GitHub/nwb-guide/src/pyflask/manageNeuroconv/manage_neuroconv.py", line 1234, in convert_all_to_nwb
    output_filepath = future.result()
  File "/Users/garrettflynn/miniconda3/envs/nwb-guide/lib/python3.9/concurrent/futures/_base.py", line 439, in result
    return self.__get_result()
  File "/Users/garrettflynn/miniconda3/envs/nwb-guide/lib/python3.9/concurrent/futures/_base.py", line 391, in __get_result
    raise self._exception
KeyError: '1d3fed0c-8278-49db-b3e8-71b8c3cbf844'

@garrettmflynn garrettmflynn self-assigned this Jun 3, 2024
@CodyCBakerPhD
Copy link
Collaborator

Using latest main branch of NeuroConv?

@garrettmflynn
Copy link
Member Author

Ah you're right. Actually the latest has a different error when using the extract_backend_configuration function:

Screenshot 2024-06-03 at 12 29 24 PM

@CodyCBakerPhD
Copy link
Collaborator

Much better - that's at least closer to the kind of error I'd expect to see now

What's the full traceback/log and are there any dumped payloads?

@garrettmflynn
Copy link
Member Author

Here's the full traceback for that error:

Traceback (most recent call last):
  File "/Users/garrettflynn/Documents/GitHub/nwb-guide/src/pyflask/manageNeuroconv/manage_neuroconv.py", line 898, in create_file
    converter.run_conversion(
  File "/Users/garrettflynn/Documents/GitHub/neuroconv/src/neuroconv/nwbconverter.py", line 229, in run_conversion
    configure_backend(nwbfile=nwbfile_out, backend_configuration=backend_configuration)
  File "/Users/garrettflynn/Documents/GitHub/neuroconv/src/neuroconv/tools/nwb_helpers/_configure_backend.py", line 31, in configure_backend
    locations_to_remap = backend_configuration.find_locations_requiring_remapping(nwbfile=nwbfile)
  File "/Users/garrettflynn/Documents/GitHub/neuroconv/src/neuroconv/tools/nwb_helpers/_configuration_models/_base_backend.py", line 99, in find_locations_requiring_remapping
    raise ValueError(
ValueError: The number of default configurations (0) does not match the number of specified configurations (15)!



127.0.0.1 - - [03/Jun/2024 12:53:41] "POST /log HTTP/1.1" 200 -

Source data is valid!
Source data is valid!
Metadata is valid!
conversion_options is valid!
Source data is valid!
Source data is valid!
Metadata is valid!
conversion_options is valid!

Source data is valid!
Source data is valid!
Metadata is valid!
conversion_options is valid!
Source data is valid!
Source data is valid!
Metadata is valid!
conversion_options is valid!

[2024-06-03 12:53:42,091] ERROR in app: Exception on /neuroconv/convert [POST]
concurrent.futures.process._RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/Users/garrettflynn/miniconda3/envs/nwb-guide/lib/python3.9/concurrent/futures/process.py", line 246, in _process_worker
    r = call_item.fn(*call_item.args, **call_item.kwargs)
  File "/Users/garrettflynn/Documents/GitHub/nwb-guide/src/pyflask/manageNeuroconv/manage_neuroconv.py", line 1128, in convert_to_nwb
    create_file(info, log_url=log_url)
  File "/Users/garrettflynn/Documents/GitHub/nwb-guide/src/pyflask/manageNeuroconv/manage_neuroconv.py", line 919, in create_file
    raise e
  File "/Users/garrettflynn/Documents/GitHub/nwb-guide/src/pyflask/manageNeuroconv/manage_neuroconv.py", line 898, in create_file
    converter.run_conversion(
  File "/Users/garrettflynn/Documents/GitHub/neuroconv/src/neuroconv/nwbconverter.py", line 229, in run_conversion
    configure_backend(nwbfile=nwbfile_out, backend_configuration=backend_configuration)
  File "/Users/garrettflynn/Documents/GitHub/neuroconv/src/neuroconv/tools/nwb_helpers/_configure_backend.py", line 31, in configure_backend
    locations_to_remap = backend_configuration.find_locations_requiring_remapping(nwbfile=nwbfile)
  File "/Users/garrettflynn/Documents/GitHub/neuroconv/src/neuroconv/tools/nwb_helpers/_configuration_models/_base_backend.py", line 99, in find_locations_requiring_remapping
    raise ValueError(
ValueError: The number of default configurations (0) does not match the number of specified configurations (15)!
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/garrettflynn/miniconda3/envs/nwb-guide/lib/python3.9/site-packages/flask/app.py", line 1484, in full_dispatch_request
    rv = self.dispatch_request()
  File "/Users/garrettflynn/miniconda3/envs/nwb-guide/lib/python3.9/site-packages/flask/app.py", line 1469, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
  File "/Users/garrettflynn/miniconda3/envs/nwb-guide/lib/python3.9/site-packages/flask_restx/api.py", line 404, in wrapper
    resp = resource(*args, **kwargs)
  File "/Users/garrettflynn/miniconda3/envs/nwb-guide/lib/python3.9/site-packages/flask/views.py", line 109, in view
    return current_app.ensure_sync(self.dispatch_request)(**kwargs)
  File "/Users/garrettflynn/miniconda3/envs/nwb-guide/lib/python3.9/site-packages/flask_restx/resource.py", line 46, in dispatch_request
    resp = meth(*args, **kwargs)
  File "/Users/garrettflynn/Documents/GitHub/nwb-guide/src/pyflask/namespaces/neuroconv.py", line 83, in post
    return convert_all_to_nwb(
  File "/Users/garrettflynn/Documents/GitHub/nwb-guide/src/pyflask/manageNeuroconv/manage_neuroconv.py", line 1201, in convert_all_to_nwb
    output_filepath = future.result()
  File "/Users/garrettflynn/miniconda3/envs/nwb-guide/lib/python3.9/concurrent/futures/_base.py", line 439, in result
    return self.__get_result()
  File "/Users/garrettflynn/miniconda3/envs/nwb-guide/lib/python3.9/concurrent/futures/_base.py", line 391, in __get_result
    raise self._exception
ValueError: The number of default configurations (0) does not match the number of specified configurations (15)!

Where would I look for the dumped payload?

@CodyCBakerPhD
Copy link
Collaborator

Where would I look for the dumped payload?

I don't know - how did you do that for the previous metadata examples?

@garrettmflynn
Copy link
Member Author

Got it. Wasn't clear you were referring to a manual dump of the payload to the log file.

Glad you mentioned it, turns out the logging feature was broken at some point—likely from the Python refactor. Unlike the global error handler, you out you have to call the logger on the part of the API it's attached to. And we were using flask_app and api interchangeably.

@CodyCBakerPhD CodyCBakerPhD mentioned this pull request Jun 3, 2024
Merged
@garrettmflynn
Copy link
Member Author

Here's a log file that appropriately catches the traceback and payload for the problematic function:

2024-06-03_13-04-49.log

@CodyCBakerPhD
Copy link
Collaborator

OK, this is going to take a bit of a deeper dive then

I say we table this for post-release and prioritize the timezone thing

@garrettmflynn
Copy link
Member Author

Gotcha sounds good.

@garrettmflynn garrettmflynn mentioned this pull request Jun 3, 2024
Merged
@CodyCBakerPhD
Copy link
Collaborator

OK I pushed another fix to the NeuroConv branch and I get a bit further

My current error is

ValueError: Chunk shape must not be greater than data shape in any dimension. (1155, 64) is not compatible with (100, 384)

which would only occur if I was trying to use the configuration with stub mode

I was able to push a fix for this and was able to run a pipeline successfully with everything, please give the NeuroConv branch another try and let me know if it works on your end

@CodyCBakerPhD
Copy link
Collaborator

Once I hear from you I think this PR can be merged into its parent

I have a few other things to comment on the main PR for this, mostly stylistic

@garrettmflynn
Copy link
Member Author

Yep this works for me! Not seeing that error on my end for some reason.

@CodyCBakerPhD
Copy link
Collaborator

Yep this works for me! Not seeing that error on my end for some reason.

Because I pushed the fix before posting that 😆

@CodyCBakerPhD CodyCBakerPhD marked this pull request as ready for review June 4, 2024 15:55
@CodyCBakerPhD CodyCBakerPhD merged commit 0024572 into backend-configuration Jun 4, 2024
7 of 15 checks passed
@CodyCBakerPhD CodyCBakerPhD deleted the backend-configuration-uniform branch June 4, 2024 15:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@CodyCBakerPhD CodyCBakerPhD CodyCBakerPhD left review comments

Assignees

@garrettmflynn garrettmflynn

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@garrettmflynn @CodyCBakerPhD