Add Conversion Progress Bars

[garrettmflynn]

This PR implements the basic skeleton for parallel conversion progress bars, specifically a parallelized global bar for conversions.

Pretty sure this still requires updates to NeuroConv to allow for within-file updates. Specifically, we need to ingest the following conversion options:

[garrettmflynn]

@CodyCBakerPhD Actually I see now in #774 that you don't think we need these options.

I'm getting this error with the above code enabled:

jsonschema.exceptions.ValidationError: Additional properties are not allowed ('iterator_opts' was unexpected)

Failed validating 'additionalProperties' in schema['properties']['Phy Sorting']:
    {'additionalProperties': False,
     'properties': {'stub_test': {'default': False,
                                  'description': 'If True, will truncate '
                                                 'the data to run the '
                                                 'conversion faster and '
                                                 'take up less memory.',
                                  'type': 'boolean'},
                    'units_description': {'default': 'Autogenerated by '
                                                     'neuroconv.',
                                          'type': 'string'},
                    'units_name': {'default': 'units',
                                   'description': 'The name of the units '
                                                  'table. If '
                                                  "write_as=='units', then "
                                                  'units_name must also be '
                                                  "'units'.",
                                   'type': 'string'},
                    'write_as': {'default': 'units',
                                 'description': 'How to save the units '
                                                'table in the nwb file. '
                                                'Options:\n'
                                                "- 'units' will save it to "
                                                'the official '
                                                'NWBFile.Units position; '
                                                'recommended only for the '
                                                'final form of the data.\n'
                                                "- 'processing' will save "
                                                'it to the processing '
                                                'module to serve as a '
                                                'historical provenance for '
                                                'the official table.',
                                 'enum': ['units', 'processing']},
                    'write_ecephys_metadata': {'default': False,
                                               'description': 'Write '
                                                              'electrode '
                                                              'information '
                                                              'contained '
                                                              'in the '
                                                              'metadata.',
                                               'type': 'boolean'}},
     'required': [],
     'type': 'object'}

On instance['Phy Sorting']:
    {'iterator_opts': {'display_progress': True,
                       'progress_bar_class': <class 'tqdm_publisher._subscriber.TQDMProgressSubscriber'>,
                       'progress_bar_options': {'mininterval': 0,
                                                'on_progress_update': <function convert_to_nwb.<locals>.update_conversion_progress at 0x125b33ca0>}},
     'stub_test': True}

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/garrettflynn/miniconda3/envs/nwb-guide/lib/python3.9/concurrent/futures/process.py", line 211, in _sendback_result
    result_queue.put(_ResultItem(work_id, result=result,
  File "/Users/garrettflynn/miniconda3/envs/nwb-guide/lib/python3.9/multiprocessing/queues.py", line 371, in put
    obj = _ForkingPickler.dumps(obj)
  File "/Users/garrettflynn/miniconda3/envs/nwb-guide/lib/python3.9/multiprocessing/reduction.py", line 51, in dumps
    cls(buf, protocol).dump(obj)
AttributeError: Can't pickle local object 'convert_to_nwb.<locals>.update_conversion_progress'
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/garrettflynn/Documents/GitHub/nwb-guide/pyflask/apis/neuroconv.py", line 117, in post
    return convert_all_to_nwb(url, **neuroconv_api.payload)
  File "/Users/garrettflynn/Documents/GitHub/nwb-guide/pyflask/manageNeuroconv/manage_neuroconv.py", line 858, in convert_all_to_nwb
    output_filepath = future.result()
  File "/Users/garrettflynn/miniconda3/envs/nwb-guide/lib/python3.9/concurrent/futures/_base.py", line 439, in result
    return self.__get_result()
  File "/Users/garrettflynn/miniconda3/envs/nwb-guide/lib/python3.9/concurrent/futures/_base.py", line 391, in __get_result
    raise self._exception
AttributeError: Can't pickle local object 'convert_to_nwb.<locals>.update_conversion_progress'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/garrettflynn/miniconda3/envs/nwb-guide/lib/python3.9/site-packages/flask/app.py", line 1484, in full_dispatch_request
    rv = self.dispatch_request()
  File "/Users/garrettflynn/miniconda3/envs/nwb-guide/lib/python3.9/site-packages/flask/app.py", line 1469, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
  File "/Users/garrettflynn/miniconda3/envs/nwb-guide/lib/python3.9/site-packages/flask_restx/api.py", line 404, in wrapper
    resp = resource(*args, **kwargs)
  File "/Users/garrettflynn/miniconda3/envs/nwb-guide/lib/python3.9/site-packages/flask/views.py", line 109, in view
    return current_app.ensure_sync(self.dispatch_request)(**kwargs)
  File "/Users/garrettflynn/miniconda3/envs/nwb-guide/lib/python3.9/site-packages/flask_restx/resource.py", line 46, in dispatch_request
    resp = meth(*args, **kwargs)
  File "/Users/garrettflynn/Documents/GitHub/nwb-guide/pyflask/apis/neuroconv.py", line 123, in post
    neuroconv_api.abort(500, str(exception))
  File "/Users/garrettflynn/miniconda3/envs/nwb-guide/lib/python3.9/site-packages/flask_restx/namespace.py", line 153, in abort
    abort(*args, **kwargs)
  File "/Users/garrettflynn/miniconda3/envs/nwb-guide/lib/python3.9/site-packages/flask_restx/errors.py", line 28, in abort
    flask.abort(code)
  File "/Users/garrettflynn/miniconda3/envs/nwb-guide/lib/python3.9/site-packages/flask/helpers.py", line 277, in abort
    current_app.aborter(code, *args, **kwargs)
  File "/Users/garrettflynn/miniconda3/envs/nwb-guide/lib/python3.9/site-packages/werkzeug/exceptions.py", line 863, in __call__
    raise self.mapping[code](*args, **kwargs)
werkzeug.exceptions.InternalServerError: 500 Internal Server Error: The server encountered an internal error and was unable to complete your request. Either the server is overloaded or there is an error in the application.

[CodyCBakerPhD]

I would not bother displaying the per file progress when running in stub mode - just the total bar should be fine (since individual files should be instantaneous)

This feature is just for non-stub full conversions with many files

[CodyCBakerPhD]

@CodyCBakerPhD Actually I see now in #774 that you don't think we need these options.

I'm getting this error with the above code enabled:

This only applies to recording and imaging interfaces; sorting, such as Phy, do not currently support it at all since they don't buffer at all (source data is small enough to load entirely to RAM)

[garrettmflynn]

Using all the open PRs together, this is working!

For some reason, there's a small delay between the completion of each file and the registration of their success on the global bar.

[garrettmflynn]

This behavior may be related to the fact that the raw TQDM updates indicate n=2 with total=1 for the final update, which still comes slightly before registration globally. Not sure why that mismatch between n and total exists

[CodyCBakerPhD]

For some reason, there's a small delay between the completion of each file and the registration of their success on the global bar.

That might just come down to an edge case on the size of this (small) data

I can try with some larger files

[CodyCBakerPhD]

@garrettmflynn it might be a while until the entire stack comes into sync with this; can you splinter out the logging improvement to a separate PR?

Also would it be possible to limit the number of error messages displayed at a time so the screen doesn't plan?

[CodyCBakerPhD]

@garrettmflynn Any idea what's up with the ubuntu pipelines here? I reran them 3 times

[garrettmflynn]

Looks like we saw this same error in #676 (comment).

Last time, we were able to track this down by looking at the artifacts. I'll do this now.

[garrettmflynn]

At least for the lastest run, it looks like the progression though one of the pipelines (previously an earlier pipeline, this time CellExplorer) freezes and the Puppeteer instance decides it's been too long after 5min.

Looking at the screenshots, everything looks good. Only weird part is that there aren't any after CellExplorer File Metadata since the stall.

[CodyCBakerPhD]

And it's happening on all of the CI now?

[garrettmflynn]

My bad, looks like I didn't wait for the final screenshot to finish before closing Puppeteer. Had to significantly refactor the iteration through all the pipeline pages so a screenshot could be injected after each—and it looks like this slipped through.

[garrettmflynn]

So I'm still not sure why this is happening, but I can at least tell you what is happening:

At a seemingly random pipeline, the tests get stuck on the stub conversion despite it finishing 3s after the page transition is triggered:

This screenshot doesn't change for another 2+ minutes until the Ubuntu test fails. This doesn't happen on Windows or Mac tests.

[CodyCBakerPhD]

Just tried this out locally on Windows using the multi-session tutorial dataset that was modified (by changing line https://github.com/NeurodataWithoutBorders/nwb-guide/blob/main/src/pyflask/manageNeuroconv/manage_neuroconv.py#L1347 from 3.0 to 100.0)

Since the number of jobs is not yet exposed to the frontend (so no parallel writing of multiple files) I would have expected only one extra bar below the main one (counting how many files remain), with the second bar tracking progress on writing the 'long' recording for each file

However, I did not see any extra bar displayed

[garrettmflynn]

Was this a full or stub conversion? Just tested on my M2 and I'm able to observe sub-bars for each file—though this is disabled for stubs.

[garrettmflynn]

While the option isn't exposed on the UI, we are telling the backend to write use 2 jobs at the moment—so you should see four sub-bars populate in groups of two.

Though to clarify, you're expecting to see one sub-bar if the n_jobs is 1?

[CodyCBakerPhD]

It was a full conversion; I don't expect sub bars for stub mode (though I would expect parallelization)

Odd, I just tried with my M1 mac and they show up fine; I'll try again on Windows later and raise an issue

Though to clarify, you're expecting to see one sub-bar if the n_jobs is 1?

Yeah, I'd always expect sub-bars per file to show how long each individual file is taking to write