Depth image from D435i

[yash7321]

No matter what filters or approaches I use, I can't seem to achieve the same depth image accuracy in Python as I do with the Intel RealSense Viewer. Is there a way to improve this? Below is the code I've written. Do you have any suggestions to make it better?

#camera.py
import pyrealsense2 as rs
import numpy as np

def initialize_camera():

pipeline = rs.pipeline()
config = rs.config()
config.enable_stream(rs.stream.depth, 640, 480, rs.format.z16, 30)
config.enable_stream(rs.stream.color, 640, 480, rs.format.bgr8, 30)
config.enable_stream(rs.stream.accel, rs.format.motion_xyz32f, 250)
config.enable_stream(rs.stream.gyro, rs.format.motion_xyz32f, 200)
profile = pipeline.start(config)

# Create an align object to align depth frames to color frames
align = rs.align(rs.stream.color)

# Create post-processing filters
decimation = rs.decimation_filter()
spatial = rs.spatial_filter()
temporal = rs.temporal_filter()
hole_filling = rs.hole_filling_filter()
threshold = rs.threshold_filter(min_dist=0.15, max_dist=7.0)

# Set filter options
spatial.set_option(rs.option.filter_magnitude, 5)
spatial.set_option(rs.option.filter_smooth_alpha, 0.75)
spatial.set_option(rs.option.filter_smooth_delta, 35)
temporal.set_option(rs.option.filter_smooth_alpha, 0.35)
temporal.set_option(rs.option.filter_smooth_delta, 45)
decimation.set_option(rs.option.filter_magnitude, 1)
hole_filling.set_option(rs.option.holes_fill, 1)

return pipeline, profile, align, decimation, spatial, temporal, hole_filling, threshold

def get_frames(pipeline, align, decimation, spatial, temporal, hole_filling, threshold, sync_tolerance=70):
frames = pipeline.wait_for_frames()
aligned_frames = align.process(frames)
depth_frame = aligned_frames.get_depth_frame()
color_frame = aligned_frames.get_color_frame()
accel_frame = frames.first_or_default(rs.stream.accel)
gyro_frame = frames.first_or_default(rs.stream.gyro)

if not depth_frame or not color_frame or not accel_frame or not gyro_frame:
    print("one or more frame missing")
    return None, None, None

color_time = color_frame.get_timestamp()
depth_time = depth_frame.get_timestamp()
accel_time = accel_frame.get_timestamp()
gyro_time = gyro_frame.get_timestamp()

if abs(depth_time - accel_time) > sync_tolerance or abs(depth_time - gyro_time) > sync_tolerance or abs(
        color_time - depth_time) > sync_tolerance:
    print("frames not in sync")
    return None, None, None

depth_frame = decimation.process(depth_frame)
# depth_frame = threshold.process(depth_frame)
disparity_transform = rs.disparity_transform(True)
depth_frame = disparity_transform.process(depth_frame)
depth_frame = spatial.process(depth_frame)
depth_frame = temporal.process(depth_frame)
depth_to_disparity = rs.disparity_transform(False)
depth_frame = depth_to_disparity.process(depth_frame)
depth_frame = hole_filling.process(depth_frame)

depth_image = np.asanyarray(depth_frame.get_data())
color_image = np.asanyarray(color_frame.get_data())
imu_data = {'accel': np.asanyarray(accel_frame.get_data()),
            'gyro': np.asanyarray(gyro_frame.get_data())}

# Check if the depth image contains valid data
if np.all(depth_image == 0):
    print("Depth image contains only zeros")
    return None, None, None

# Debug print the depth and color images
print(f"Depth image shape: {depth_image.shape}, dtype: {depth_image.dtype}")
print(f"Color image shape: {color_image.shape}, dtype: {color_image.dtype}")
print(f"Depth image sample: {depth_image[:5, :5]}")
print(f"Color image sample: {color_image[:5, :5, :]}")

return depth_image, color_image, imu_data

[MartyG-RealSense]

Hi @yash7321 Does accuracy noticably improve if you use 848x480 for the depth and color resolution instead of 640x480? 848x480 is the optimal depth resolution for accuracy for most RealSense 400 Series cameras (on the D415 model 1280x720 is the optimal resolution).

The Hole-Filling post-processing filter is disabled by default in the RealSense Viewer, so please also try disabling that filter to see whether your results in Python are closer to those in the Viewer.

Intel's guidelines for the order in which to apply post-processing filters in recommends that depth_to_disparity is applied before the Spatial filter.

https://dev.intelrealsense.com/docs/post-processing-filters#using-filters-in-application-code

It is also recommended that alignment is applied after the list of post-processing 'process' lines, whilst in your script alignment is applied before the post-processing filters are applied.

[yash7321]

the depth image i get is wrt to which exact point on camera , after aligning depth to color image,
based on camera dimensions ,?

[yash7321]

and also does this mean my camera is in upright position?

[MartyG-RealSense]

Your camera is in an upright position if the side with the large single central screw-thread hole (where the tripod attached) is facing straight downwards towards the ground.

Yes, the IMU axes in the above image represent the upright position, facing forwards.

After aligning depth to color, the 0,0,0 origin of depth changes from the center-line of the left infrared sensor to the center-line of the RGB sensor.

[yash7321]

cam you tell me exactly how camera and imu axis are aligned, i have gone through files but still cofused

Camera Coordinate System (Intel RealSense):

X-axis: Right
Y-axis: Down
Z-axis: Forward

IMU Coordinate System (Intel RealSense D435i):

Accelerometer:
when in front me not facing me but opposite(same as where i am looking imu is looking there)

X-axis: left
Y-axis: down
Z-axis: towards me

Gyroscope:X-axis: Pitch (rotation around X-axis)
Y-axis: Yaw (rotation around Y-axis)
Z-axis: Roll (rotation around Z-axis)

is this correct?

[MartyG-RealSense]

Camera coordinate system: The positive x-axis points to the right. The positive y-axis points down. The positive z-axis points forward. So you are correct about these camera axes.

IMU coordinate system: Acceleration uses the same vectors as the camera coordinate system (x-right, y-down, z-forward).

The arrows on the diagram below illustrate the directions of the gyroscopic x, y and z angles.