You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
With sensors that acquire large amounts of data (>~ MB), the TCP network transfer becomes a bottleneck on acquisition. Writing to file would be a much more performant method.
I want to formally discuss the prospect of a new trait (writes-measured or similar) that allows sensors to pass on references to data, rather than the data itself. The trait would have a new message (get_measured_reference or similar) to retrieve the reference.
Bluesky supports such references, so we should introduce a trait that works well in their system. From what I gather, a single file (let's say hdf5 format) is used for an acquisition, and each measurement Events point to a Datum document that specifies how to extract the specific measurement from the entire file. So a measured_filepath property, with a setter and getter, seem necessary for this trait. The trait's data reference will likely be some form of dictionary with information on how to lookup the data within the file.
For the purposes of streaming data (i.e. not doing an acquisition, but perhaps viewing data live for alignment), it would be desirable to have some option for a writing mode that buffers the last N acquisitions in a file so SMWR can be used to poll the latest element (assuming SWMR works well enough). The buffer is desirable to avoid using up disc space when not needed. Another (easier) option is to introduce a property to turn on/off writing.
The text was updated successfully, but these errors were encountered:
With sensors that acquire large amounts of data (>~ MB), the TCP network transfer becomes a bottleneck on acquisition. Writing to file would be a much more performant method.
I want to formally discuss the prospect of a new trait (
writes-measured
or similar) that allows sensors to pass on references to data, rather than the data itself. The trait would have a new message (get_measured_reference
or similar) to retrieve the reference.Bluesky supports such references, so we should introduce a trait that works well in their system. From what I gather, a single file (let's say hdf5 format) is used for an acquisition, and each measurement Events point to a Datum document that specifies how to extract the specific measurement from the entire file. So a
measured_filepath
property, with a setter and getter, seem necessary for this trait. The trait's data reference will likely be some form of dictionary with information on how to lookup the data within the file.For the purposes of streaming data (i.e. not doing an acquisition, but perhaps viewing data live for alignment), it would be desirable to have some option for a writing mode that buffers the last N acquisitions in a file so SMWR can be used to poll the latest element (assuming SWMR works well enough). The buffer is desirable to avoid using up disc space when not needed. Another (easier) option is to introduce a property to turn on/off writing.
The text was updated successfully, but these errors were encountered: