OrbbecSDK-K4A-Wrapper-1.9.3: transformation_depth_image_to_color_camera_validate_parameters - wrapper fails

[Bravehawk666666]

Hi!

we are trying to replace our Azure Kinects with Femto Bolts.

We use pyk4a as an easy to use python interface to connect to the kinects. Code is running since months with the kinects without problems.

After installing the Femto Bolt wrapper (copy dlls and libs into ..\Program... \Azure Kinect SDK v1.4.1\sdk\windows-desktop\amd64\release\lib and ...\bin) the standard Azure Kinect Viewer is working with the Femto Bolt (except (sometimes) some latency between color/depth - but this seems to be another issue). For now: lets say this works.

Now we try our python script to get the registered images (color2depth). This is a stripped down code:

import numpy as np
import pyk4a
from pyk4a import Config, PyK4A
    
def main():    
    k4a = PyK4A(
            Config(
                color_resolution=pyk4a.ColorResolution.RES_720P,
                depth_mode=pyk4a.DepthMode.NFOV_UNBINNED,
                synchronized_images_only=True,
                camera_fps=pyk4a.FPS.FPS_30
            )
        )
    k4a.start()

    while True:
        # Capture a frame
        capture = k4a.get_capture()

        # Get the depth and color images
        depth_image = capture.depth
        color_image = capture.color
        
        depth_image_transformed = capture.transformed_depth ## <<<< femto bolt (wrapper?) problem
        
        #... do stuff with transformed image
      
    k4a.stop()

if __name__ == "__main__":
    main()

... and this is the output:

python .\hello_femto_debug.py
[04/08 14:47:19.393140][info][26940][Context.cpp:69] Context created with config: default config!
[04/08 14:47:19.393382][info][26940][Context.cpp:74] Context work_dir=C:\Users\Manuel\OneDrive\Documents\prg\hello_chatty_experiments
[04/08 14:47:19.394058][info][26940][Context.cpp:77]    - SDK version: 1.9.5
[04/08 14:47:19.394337][info][26940][Context.cpp:78]    - SDK stage version: main
[04/08 14:47:19.394605][info][26940][Context.cpp:82] get config EnumerateNetDevice:false
[04/08 14:47:19.395026][info][26940][MfPal.cpp:102] createObPal: create WinPal!
[04/08 14:47:19.426212][info][26940][DeviceManager.cpp:15] Current found device(s): (1)
[04/08 14:47:19.426315][info][26940][DeviceManager.cpp:24]      - Name: Femto Bolt, PID: 0x066b, SN/ID: xxxx, Connection: USB3.1
[04/08 14:47:19.427370][info][26940][FemtoBoltUvcDevice.cpp:23] FemtoBoltUvcDevice init ...
[04/08 14:47:19.427507][info][26940][FemtoBoltUvcDevice.cpp:120] Create command start!
[04/08 14:47:19.427729][info][26940][MfPal.cpp:232] Create MSDEConverterDevice uvc device.
[04/08 14:47:19.430585][info][26940][MSDEConverterDevice.cpp:721] Succeed to load depth engine plugin
[04/08 14:47:20.324353][info][26940][FemtoBoltUvcDevice.cpp:271] Create command done!
[04/08 14:47:20.324496][info][26940][FemtoBoltUvcDevice.cpp:431] init sensor map start!
[04/08 14:47:20.325191][info][26940][FemtoBoltUvcDevice.cpp:458] init sensor map done!
[04/08 14:47:20.327352][info][26940][AbstractDevice.cpp:124]    - Firmware version: 1.1.0
[04/08 14:47:20.329287][info][26940][FemtoBoltUvcDevice.cpp:275] Init depth process param start!
[04/08 14:47:20.340977][info][26940][MSDEConverterDevice.cpp:772] got nvram data succeed.
[04/08 14:47:20.987099][info][26940][FemtoBoltUvcDevice.cpp:402] setNvramDataStreamStopFunc succeed
[04/08 14:47:20.988581][info][26940][FemtoBoltUvcDevice.cpp:427] Init depth process param done!
[04/08 14:47:21.359118][info][26940][FemtoBoltUvcDevice.cpp:38] FemtoBoltUvcDevice init done!
[04/08 14:47:21.359273][info][26940][DeviceManager.cpp:157] Device created successfully! Name: Femto Bolt, PID: 0x066b, SN/ID: xxxx
[04/08 14:47:21.360252][info][26940][Pipeline.cpp:44] Pipeline created with device: {name: Femto Bolt, sn: xxxx}, @0x1E17EF021B0
[04/08 14:47:21.366101][info][26940][Pipeline.cpp:678] config is nullptr,return default calibration param!
[04/08 14:47:21.374333][info][26940][FemtoBoltUvcDevice.cpp:554] Depth sensor has been created!
[04/08 14:47:21.374738][info][26940][FemtoBoltUvcDevice.cpp:619] Ir sensor has been created!
[04/08 14:47:21.379480][info][26940][FemtoBoltUvcDevice.cpp:585] Color sensor has been created!
[04/08 14:47:21.726211][info][26940][Pipeline.cpp:239] Try to start streams!
[04/08 14:47:21.726523][info][26940][VideoSensor.cpp:663] start OB_SENSOR_DEPTH stream with profile: {type: OB_STREAM_DEPTH, format: OB_FORMAT_Y16, width: 640, height: 576, fps: 30}
[04/08 14:47:21.727650][info][26940][MSDEConverterDevice.cpp:544] Start real profile,width:7680 height:434
[04/08 14:47:21.731965][info][42152][MSDEConverterDevice.cpp:75] Depth engine got nvram data size:495208
[04/08 14:47:21.732074][info][42152][MSDEConverterDevice.cpp:102] use dynlib load depthengine lib......
[04/08 14:47:21.915081][info][42152][MSDEConverterDevice.cpp:113] Depth engine init succeed!
[04/08 14:47:22.216324][info][26940][VideoSensor.cpp:663] start OB_SENSOR_IR stream with profile: {type: OB_STREAM_IR, format: OB_FORMAT_Y16, width: 640, height: 576, fps: 30}
[04/08 14:47:22.216945][info][26940][VideoSensor.cpp:663] start OB_SENSOR_COLOR stream with profile: {type: OB_STREAM_COLOR, format: OB_FORMAT_BGRA, width: 1280, height: 720, fps: 30}
[04/08 14:47:22.273896][info][26940][Pipeline.cpp:252] Start streams done!
[04/08 14:47:22.273971][info][26940][Pipeline.cpp:235] Pipeline start done!
[04/08 14:47:22.276113][error][26940][IMUFrameReversion.cpp:8] entry IMUFrameReversion
[2024-04-08 14:47:22.276] [error] [t=26940] C:\Users\daiyin\Desktop\OrbbecSDK-K4A-Wrapper\src\orbbec\ob_type_helper.cpp (47): orbbec_sdk_log(). [04/08 14:47:22.276113][error][26940][IMUFrameReversion.cpp:8] entry IMUFrameReversion

[04/08 14:47:22.276453][info][26940][AccelSensor.cpp:26] AccelSensor created
[04/08 14:47:22.276673][info][26940][FemtoBoltUvcDevice.cpp:681] Accel sensor has been created!
[04/08 14:47:22.586026][error][26940][IMUFrameReversion.cpp:8] entry IMUFrameReversion
[2024-04-08 14:47:22.586] [error] [t=26940] C:\Users\daiyin\Desktop\OrbbecSDK-K4A-Wrapper\src\orbbec\ob_type_helper.cpp (47): orbbec_sdk_log(). [04/08 14:47:22.586026][error][26940][IMUFrameReversion.cpp:8] entry IMUFrameReversion

[04/08 14:47:22.587439][info][26940][GyroSensor.cpp:27] GyroSensor created!
[04/08 14:47:22.587544][info][26940][FemtoBoltUvcDevice.cpp:723] Gyro sensor has been created!


[2024-04-08 14:47:23.031] [error] [t=26940] C:\Users\daiyin\Desktop\OrbbecSDK-K4A-Wrapper\src\transformation\rgbz.c (113): transformation_compare_image_descriptors(). Unexpected image descriptor. Descriptor 1: width_pixels: 640, height_pixels: 576, stride_bytes: 1280, format: 7. Descriptor 2: width_pixels: 640, height_pixels: 576, stride_bytes: 1280, format: 4.


[2024-04-08 14:47:23.031] [error] [t=26940] C:\Users\daiyin\Desktop\OrbbecSDK-K4A-Wrapper\src\transformation\rgbz.c (733): transformation_depth_image_to_color_camera_validate_parameters(). Unexpected depth image descriptor, see details above.
[2024-04-08 14:47:23.032] [error] [t=26940] C:\Users\daiyin\Desktop\OrbbecSDK-K4A-Wrapper\src\transformation\transformation.c (710): transformation_depth_image_to_color_camera_validate_parameters( &transformation_context->calibration, &transformation_context->depth_camera_xy_tables, depth_image_data, depth_image_descriptor, custom_image_data, custom_image_descriptor, transformed_depth_image_data, transformed_depth_image_descriptor, transformed_custom_image_data, transformed_custom_image_descriptor) returned failure in transformation_depth_image_to_color_camera_custom()
[2024-04-08 14:47:23.032] [error] [t=26940] C:\Users\daiyin\Desktop\OrbbecSDK-K4A-Wrapper\src\orbbec\ob_k4a_impl.cpp (3799): transformation_depth_image_to_color_camera_custom(transformation_handle, depth_image_buffer, &depth_image_descriptor, custom_image_buffer, &dummy_descriptor, transformed_depth_image_buffer, &transformed_depth_image_descriptor, transformed_custom_image_buffer, &dummy_descriptor, interpolation_type, invalid_custom_value) returned failure in k4a_transformation_depth_image_to_color_camera()

So in rgbz.c the error because of the type mismatch occurs:

rgbz.c (113): transformation_compare_image_descriptors(). Unexpected image descriptor.
Descriptor 1: width_pixels: 640, height_pixels: 576, stride_bytes: 1280, format: 7.
Descriptor 2: width_pixels: 640, height_pixels: 576, stride_bytes: 1280, format: 4.

It seems to be a mismatch of the format between K4A_IMAGE_FORMAT_CUSTOM16 and K4A_IMAGE_FORMAT_DEPTH16. I naively tried to fix it by hardcoding depth16 in the wrapper but it seems to be a bit more complex than that.

Used versions:
pyk4a Version 1.5 (https://github.com/etiennedub/pyk4a/releases/tag/1.5.0),
python 3.9.16
orbbecsdk-k4a-wrapper: 1.93
windows 10

Thanks for any help with this!