Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

High frequency inputs to NeuropixelsV2eBno055Data node reveals multithreading issue during workflow restart #192

Open
jvoigts opened this issue Aug 3, 2024 · 5 comments
Assignees
Labels
bug Something isn't working
Milestone

Comments

@jvoigts
Copy link
Member

jvoigts commented Aug 3, 2024

This can occur either when using the restart button in bonsai or stopping and starting rapidly (within about a second maybe).
I'm seeing this with two V2e neuropixels, nothing else in the workflow. I have not properly tried different workflows etc. yet, so I have no info on whether this is specific to this configuration.

@jvoigts jvoigts added the bug Something isn't working label Aug 3, 2024
@jonnew jonnew changed the title Rapid restarting of a NeuropixelsV2e workflow gives 'attempt to trigger already triggered operation' error. High frequency inputs to NeuropixelsV2eBno055Data node reveals multithreading issue Aug 4, 2024
@jonnew
Copy link
Member

jonnew commented Aug 4, 2024

If a NeuropixelsV2eBno055Data is feed with a high-frequency sequence, the user will very likely be presented with an "Attempt to trigger and already triggered operation" exception. Two issues:

  • This error string is really uninformative. What it means is that there is an ongoing read or write to a device register when another is attempted. Issue to improve these strings here: Review error strings liboni#26
  • The presence of the error reveals a threading issue that is probably always present in this library but has a high probability of occurring when NeuropixelsV2eBno055Data (and nodes that behave like it) are feed with a high-frequency sequence. We found this issue by feeding the NeuropixelsV2eBno055Data` with the output of `NeuropixelsV2eBnoData` which is produced at ~30 kHz. What seems to be happening is that NeuropixelsV2eBno055Data` is attempting to access the configuration channel to read a BNO value while its being used to for the disposal processes that occurring during a workflow restart. Obviously this is not an advisable topology but its valid so we need to fix.

Side note: I would attach a workflow but I dont know how because of githubs file type filters.

@jonnew jonnew added this to the 0.2.0 milestone Aug 4, 2024
@glopesdev
Copy link
Collaborator

glopesdev commented Aug 4, 2024

@jonnew I think the issue that @jvoigts raised is a separate one from this new polling issue that you raised, and it is a known issue with asynchronous dispose (i.e. the previous context may be disposing while a new one is being created) that I was still considering how to solve.

I think the high frequency polling issue is different even though they output the same low-level oni exception, so I would open two separate issues until we conclude definitively that they are the same.

@jonnew
Copy link
Member

jonnew commented Aug 4, 2024

I dont think so -- we worked together to figure out what would fix the original issue he raised and this was the cause

@glopesdev
Copy link
Collaborator

@jonnew ok, just to confirm since I am realizing your original comment did not fully clarify, when is the error exactly thrown? Is it still when the user starts and stops the workflow? Or an error during polling at runtime?

If the former then I agree it might be the same error.

@jonnew
Copy link
Member

jonnew commented Aug 5, 2024

I should have made I clearer (it's the 2nd to last sentence of the second bullet) that this occurs during a workflow restart. Without a high frequency input to the NeuropixelsV2eBno055Data node the workflow can be restarted without issue.

@jonnew jonnew changed the title High frequency inputs to NeuropixelsV2eBno055Data node reveals multithreading issue High frequency inputs to NeuropixelsV2eBno055Data node reveals multithreading issue during workflow restart Aug 5, 2024
@jonnew jonnew modified the milestones: 0.2.0, 0.3.0 Aug 13, 2024
@jonnew jonnew modified the milestones: 0.3.0, 0.4.0 Aug 26, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

3 participants