[Bug] Pass customized backend to run_conversion

[CodyCBakerPhD]

attn @garrettmflynn

Can you give this a try for NeurodataWithoutBorders/nwb-guide#821?

[garrettmflynn]

Getting a few different errors with this.

If I try to set the values on the configuration object, I get this:

[2024-06-03 13:17:10,597] ERROR in app: Exception on /neuroconv/configuration [POST]
Traceback (most recent call last):
  File "/Users/garrettflynn/miniconda3/envs/nwb-guide/lib/python3.9/site-packages/flask/app.py", line 1484, in full_dispatch_request
    rv = self.dispatch_request()
  File "/Users/garrettflynn/miniconda3/envs/nwb-guide/lib/python3.9/site-packages/flask/app.py", line 1469, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
  File "/Users/garrettflynn/miniconda3/envs/nwb-guide/lib/python3.9/site-packages/flask_restx/api.py", line 404, in wrapper
    resp = resource(*args, **kwargs)
  File "/Users/garrettflynn/miniconda3/envs/nwb-guide/lib/python3.9/site-packages/flask/views.py", line 109, in view
    return current_app.ensure_sync(self.dispatch_request)(**kwargs)
  File "/Users/garrettflynn/miniconda3/envs/nwb-guide/lib/python3.9/site-packages/flask_restx/resource.py", line 46, in dispatch_request
    resp = meth(*args, **kwargs)
  File "/Users/garrettflynn/Documents/GitHub/nwb-guide/src/pyflask/namespaces/neuroconv.py", line 101, in post
    return get_backend_configuration(neuroconv_namespace.payload)
  File "/Users/garrettflynn/Documents/GitHub/nwb-guide/src/pyflask/manageNeuroconv/manage_neuroconv.py", line 966, in get_backend_configuration
    configuration = update_backend_configuration(info)
  File "/Users/garrettflynn/Documents/GitHub/nwb-guide/src/pyflask/manageNeuroconv/manage_neuroconv.py", line 946, in update_backend_configuration
    backend_configuration.dataset_configurations[dataset_name][key] = value
TypeError: 'HDF5DatasetIOConfiguration' object does not support item assignment

And if I comment this out, I observe an error when creating some data on the stub conversion step—one that doesn't occur without the update_backend_configuration extraction of the configuration object:

[2024-06-03 13:14:18,050] ERROR in app: Exception on /neuroconv/convert [POST]
concurrent.futures.process._RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/Users/garrettflynn/Documents/GitHub/hdmf/src/hdmf/backends/hdf5/h5tools.py", line 1376, in __setup_chunked_dset__
    dset = parent.create_dataset(name, **io_settings)
  File "/Users/garrettflynn/miniconda3/envs/nwb-guide/lib/python3.9/site-packages/h5py/_hl/group.py", line 183, in create_dataset
    dsid = dataset.make_new_dset(group, shape, dtype, data, name, **kwds)
  File "/Users/garrettflynn/miniconda3/envs/nwb-guide/lib/python3.9/site-packages/h5py/_hl/dataset.py", line 72, in make_new_dset
    raise ValueError(errmsg)
ValueError: Chunk shape must not be greater than data shape in any dimension. (78125, 64) is not compatible with (100, 384)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/garrettflynn/miniconda3/envs/nwb-guide/lib/python3.9/concurrent/futures/process.py", line 246, in _process_worker
    r = call_item.fn(*call_item.args, **call_item.kwargs)
  File "/Users/garrettflynn/Documents/GitHub/nwb-guide/src/pyflask/manageNeuroconv/manage_neuroconv.py", line 1129, in convert_to_nwb
    create_file(info, log_url=log_url)
  File "/Users/garrettflynn/Documents/GitHub/nwb-guide/src/pyflask/manageNeuroconv/manage_neuroconv.py", line 920, in create_file
    raise e
  File "/Users/garrettflynn/Documents/GitHub/nwb-guide/src/pyflask/manageNeuroconv/manage_neuroconv.py", line 899, in create_file
    converter.run_conversion(
  File "/Users/garrettflynn/Documents/GitHub/neuroconv/src/neuroconv/nwbconverter.py", line 228, in run_conversion
    configure_backend(nwbfile=nwbfile_out, backend_configuration=backend_configuration)
  File "/Users/garrettflynn/miniconda3/envs/nwb-guide/lib/python3.9/contextlib.py", line 126, in __exit__
    next(self.gen)
  File "/Users/garrettflynn/Documents/GitHub/neuroconv/src/neuroconv/tools/nwb_helpers/_metadata_and_file_helpers.py", line 225, in make_or_load_nwbfile
    io.write(nwbfile)
  File "/Users/garrettflynn/Documents/GitHub/hdmf/src/hdmf/utils.py", line 668, in func_call
    return func(args[0], **pargs)
  File "/Users/garrettflynn/Documents/GitHub/hdmf/src/hdmf/backends/hdf5/h5tools.py", line 376, in write
    super().write(**kwargs)
  File "/Users/garrettflynn/Documents/GitHub/hdmf/src/hdmf/utils.py", line 668, in func_call
    return func(args[0], **pargs)
  File "/Users/garrettflynn/Documents/GitHub/hdmf/src/hdmf/backends/io.py", line 99, in write
    self.write_builder(f_builder, **kwargs)
  File "/Users/garrettflynn/Documents/GitHub/hdmf/src/hdmf/utils.py", line 668, in func_call
    return func(args[0], **pargs)
  File "/Users/garrettflynn/Documents/GitHub/hdmf/src/hdmf/backends/hdf5/h5tools.py", line 814, in write_builder
    self.write_group(self.__file, gbldr, **kwargs)
  File "/Users/garrettflynn/Documents/GitHub/hdmf/src/hdmf/utils.py", line 668, in func_call
    return func(args[0], **pargs)
  File "/Users/garrettflynn/Documents/GitHub/hdmf/src/hdmf/backends/hdf5/h5tools.py", line 996, in write_group
    self.write_group(group, sub_builder, **kwargs)
  File "/Users/garrettflynn/Documents/GitHub/hdmf/src/hdmf/utils.py", line 668, in func_call
    return func(args[0], **pargs)
  File "/Users/garrettflynn/Documents/GitHub/hdmf/src/hdmf/backends/hdf5/h5tools.py", line 1001, in write_group
    self.write_dataset(group, sub_builder, **kwargs)
  File "/Users/garrettflynn/Documents/GitHub/hdmf/src/hdmf/utils.py", line 668, in func_call
    return func(args[0], **pargs)
  File "/Users/garrettflynn/Documents/GitHub/hdmf/src/hdmf/backends/hdf5/h5tools.py", line 1300, in write_dataset
    dset = self.__setup_chunked_dset__(parent, name, data, options)
  File "/Users/garrettflynn/Documents/GitHub/hdmf/src/hdmf/backends/hdf5/h5tools.py", line 1378, in __setup_chunked_dset__
    raise Exception("Could not create dataset %s in %s" % (name, parent.name)) from exc
Exception: Could not create dataset data in /acquisition/ElectricalSeriesLF
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/garrettflynn/miniconda3/envs/nwb-guide/lib/python3.9/site-packages/flask/app.py", line 1484, in full_dispatch_request
    rv = self.dispatch_request()
  File "/Users/garrettflynn/miniconda3/envs/nwb-guide/lib/python3.9/site-packages/flask/app.py", line 1469, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
  File "/Users/garrettflynn/miniconda3/envs/nwb-guide/lib/python3.9/site-packages/flask_restx/api.py", line 404, in wrapper
    resp = resource(*args, **kwargs)
  File "/Users/garrettflynn/miniconda3/envs/nwb-guide/lib/python3.9/site-packages/flask/views.py", line 109, in view
    return current_app.ensure_sync(self.dispatch_request)(**kwargs)
  File "/Users/garrettflynn/miniconda3/envs/nwb-guide/lib/python3.9/site-packages/flask_restx/resource.py", line 46, in dispatch_request
    resp = meth(*args, **kwargs)
  File "/Users/garrettflynn/Documents/GitHub/nwb-guide/src/pyflask/namespaces/neuroconv.py", line 83, in post
    return convert_all_to_nwb(
  File "/Users/garrettflynn/Documents/GitHub/nwb-guide/src/pyflask/manageNeuroconv/manage_neuroconv.py", line 1202, in convert_all_to_nwb
    output_filepath = future.result()
  File "/Users/garrettflynn/miniconda3/envs/nwb-guide/lib/python3.9/concurrent/futures/_base.py", line 439, in result
    return self.__get_result()
  File "/Users/garrettflynn/miniconda3/envs/nwb-guide/lib/python3.9/concurrent/futures/_base.py", line 391, in __get_result
    raise self._exception
Exception: Could not create dataset data in /acquisition/ElectricalSeriesLF

Where I simply updated this to have BZip compression on the frontend. This looks more like a real error?

[CodyCBakerPhD]

If I try to set the values on the configuration object, I get this:

Ah, yeah. Sorry, my bad - Pydantic does need the strattr instead

Please pull that branch of GUIDE and try again

[garrettmflynn]

Alright same error as the original log file sent

The number of default configurations (0) does not match the number of specified configurations (15)

[CodyCBakerPhD]

@garrettmflynn Will need approval to merge and cut a new release to go along with the GUIDE

[garrettmflynn]

Works on my end. Thanks for pushing on this!

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 91.81%. Comparing base (aeb3dd9) to head (c221b52).
Report is 118 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #885      +/-   ##
==========================================
+ Coverage   91.70%   91.81%   +0.10%     
==========================================
  Files         125      125              
  Lines        6911     7035     +124     
==========================================
+ Hits         6338     6459     +121     
- Misses        573      576       +3     

Flag	Coverage Δ
unittests	91.81% <100.00%> (+0.10%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
src/neuroconv/basedatainterface.py	96.82% <100.00%> (+0.10%)	⬆️
src/neuroconv/nwbconverter.py	96.74% <100.00%> (+0.05%)	⬆️