Implementing Async Inference Pipeline Within Non-openvino Models Inference Loop With C++

[Spawnfile]

Hi all,

I've got 2 models which runs sequentially. One of them is detecting objects which is a non-openvino model, and the other one is a openvino public classifier model which runs on crops that generated from detection model. I'm able to implement openvino detection model which located at the top of network but I'm struggling to make the inference Async inside of the another inference pipeline. How can I implement my classifier openvino model inside other networks inference pipeline with C++ (Python snippet works for me as well) ?

Here is a python pseudo-code for better pipeline understanding,

# video loop and getting frames
detections = non_openvino_model(inputs)

for detection in detections:
    # generating crops from detections to feed them into classifier
    crops = ...
    
    # Async openvino classifier starts
    # here is where I'm stuck

Any help is appreciated
Greetings

[akuporos]

Hello @Spawnfile,

I think, you may try to call openvino inference in the callback, which is passed to the first inference function.
Something like this:

from openvino import AsyncInferQueue

infer_queue = AsyncInferQueue(compiled_model, num_request)
def callback():
    infer_queue.start_async()
another_infer(data, callback)

You may check AsyncInferQueue API here.
AsyncInferQueue available only in Python API.

[Spawnfile]

Sorry for late response, this looks promising for me thank you. I'll give a try asap. In C++ is there anything which equals to AsyncInferQueue ?

[rkazants]

Hi @Spawnfile,

Please check samples with async:

• https://github.com/openvinotoolkit/openvino/tree/master/samples/cpp/classification_sample_async
• https://github.com/openvinotoolkit/openvino/tree/master/samples/python/classification_sample_async

Best regards,
Roman

[hegdeadithyak]

Have a look at this @Spawnfile : https://gist.github.com/dkurt/59a7e7b5b45f9c46b31e01ce47d098a2