How to take accurate Frame snapshot with multiple cameras using C#

[dwventer]

I have a c# WPF project and I'm trying to take a snapshot of incoming Frames when a button is pressed. So the idea is that when the user presses the button on the app that exact Frame is presented to the user from when the button was pressed. There can be 4 or more cameras connected via a USB port.

My problem is that it looks like there is an unpredictable lag between when the button is pressed and when the desired Frames are produced. I have tried numerous methods, all of which seem to suffer the same unpredictable lag. With lag, I'm referring to the fact that it can take 3, 4 or even 11 frames before the desired Frames are received.

To compound the issue, when I retrieve a FrameSet, the colour frame and IR frames are not always synchronized.

I use the following configuration to start my Camera stream:

_deviceConfig.EnableDevice(SerialNumber);
_deviceConfig.EnableStream(Stream.Infrared, stream_index: 1, 1920, 1080, Format.Y8, 15);
_deviceConfig.EnableStream(Stream.Infrared, stream_index: 2, 1920, 1080, Format.Y8, 15);
_deviceConfig.EnableStream(Stream.Color, 1920, AspectRatio.Height, Format.Rgb8, 15);

As I said before, I have tried numerous methods. This is the latest code:

public void StartStream(CancellationToken cancellationToken)
{

    _pipelineTask = Task.Run(() => {
        PipelineProfile profile = _pipeline.Start(_deviceConfig); 
        RunPipeline(cancellationToken); });  
}

private void RunPipeline(CancellationToken cancellationToken)
{
    try
    {
        TimeSpan delay = new();
        while (!cancellationToken.IsCancellationRequested)
        {
            //using (FrameSet frames = _pipeline.WaitForFrames())
            if (_pipeline.TryWaitForFrames(out FrameSet frames, 200))
            {
		Application.Current.Dispatcher.Invoke(DispatcherPriority.Render, TimeSpan.FromMicroseconds(100), UpdateImage, frames);  
            }
        }
        //Button has been pressed
        ... Do something to get the specific frame
        _pipeline.Stop();
	//Present the Frame in Question
    }
    catch (Exception ex)
    {
        Trace.WriteLine($"Error in RunPipeline: {ex.Message}.");
    }
}

I'm using Intel RealSense D415 cameras. Can anybody, provide me with an acceptable solution?

[MartyG-RealSense]

Hi @dwventer With multiple camera applications it is recommended to use poll_for_frames() instead of wait_for_frames(), which is better suited to single-camera applications. More information about this can be found at #2422 (comment)

[dwventer]

Hi @MartyG-RealSense , Like I said. I tried many many options. PollForFrames (and even TryWaitForFrames) gives me horrible results, even when adding a sleep method. The only one that gives me good video feedback is just the simple WaitForFrames.

My problem remains, how do I get my desired Frame?

[MartyG-RealSense]

A RealSense user who created in C++ a script at #2219 (comment) that captured from all attached cameras simultaneously took the approach of using threads. There is very limited information and code references available regarding implementing multi-threading in C# though, with one of the best available being the discussion at #8334

[dwventer]

Is there a way one can use the Timestamp and/or TimeOfArrival to determine the desired frame from a 'DateTime.Now' (taken on the Triggered event)?

[MartyG-RealSense]

THe only C# script I know of that uses DateTime.Now in relation to timestamps is the example at the link below, though it may not be what you are looking for.

https://github.com/IntelRealSense/librealsense/blob/master/wrappers/csharp/Documentation/cookbook.md#playback

var t = Task.Run(async () =>
    {
        var start = DateTime.Now;
        void Print() => Console.WriteLine($"Real: {DateTime.Now - start} Status: {playback.Status,-7} Playback: {playback.Position.ToTimeSpan()}/{playback.Duration.ToTimeSpan()}");
        while (playback.Status != PlaybackStatus.Stopped)
        {
            Print();
            if (pipe.PollForFrames(out FrameSet frames))
                using (frames)
                {
                    foreach (var f in frames)
                        using (f)
                        using (var p = f.Profile)
                            Console.WriteLine($"    {p.Stream} {p.Format,4} #{f.Number} {f.TimestampDomain} {f.Timestamp:F2}");
                }