您好作者，我本次使用D455深度相机，跑您的开源代码H2-Mapping，有以下问题想和您请教一下 #45
Comments
|
图1、图2，是您之前给的可视化窗口预览的图像 还有就是，您跑这个建图时，建图的精度能达到多少呢？ |
|
A：分辨率调整后需要同步修改tracking和mapping部分的相机内参 B:这个方法对深度和pose精度要求还是比较高的 及 size其实并没有使用，如果需要切分，可以改一下代码 |
|
好的 十分感谢 |
您好博主，这段时间我购买了L515深度相机进行测试，感觉效果还可以。
|
|
您好，可以参考Readme中的“Note: The default configuration was utilized in our paper's experiment, conducted on NVIDIA Jetson AGX Orin (32GB RAM). In case you encounter memory limitations on your platform, you can attempt reducing the insert_ratio parameter in the configs/realsense/tower.yaml file, but it may result in inferior outcomes.”。或者可以使用H3-Mapping(https://github.com/SYSU-STAR/H3-Mapping)中的 adaptive_kf_pruning功能
[https://opengraph.githubassets.com/1bb9e08b76683a07e6ce70d2a650b0ff2a9b0bf1cc52029afc3c0412ac406e6d/SYSU-STAR/H3-Mapping]<https://github.com/SYSU-STAR/H3-Mapping>
GitHub - SYSU-STAR/H3-Mapping: H3-Mapping: Quasi-Heterogeneous Feature Grids for Real-time Dense Mapping Using Hierarchical Hybrid Representation (submitted to RAL 2024)<https://github.com/SYSU-STAR/H3-Mapping>
H3-Mapping: Quasi-Heterogeneous Feature Grids for Real-time Dense Mapping Using Hierarchical Hybrid Representation (submitted to RAL 2024) - SYSU-STAR/H3-Mapping
github.com
…________________________________
发件人: DeBinW ***@***.***>
发送时间: 2024年8月3日 20:33
收件人: SYSU-STAR/H2-Mapping ***@***.***>
抄送: JIANG Chenxing ***@***.***>; Comment ***@***.***>
主题: Re: [SYSU-STAR/H2-Mapping] 您好作者，我本次使用D455深度相机，跑您的开源代码H2-Mapping，有以下问题想和您请教一下 (Issue #45)
您好博主，这段时间我购买了L515深度相机进行测试，感觉效果还可以。
但是又遇到了运行时长不满足扫描一整个屋子的情况（运行到800多帧时，就会报错GPU已满）。
于是我过来向您请教一下：
在代码层面是不是可以进行优化呢？您在使用时有进行过对于代码的优化吗？
能否麻烦您提供一个思路或者有相关代码的话可不可以进行分享^_^
―
Reply to this email directly, view it on GitHub<#45 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/APMKKNQAG7XZTTDGSTDNLHLZPTEYVAVCNFSM6AAAAABLN7TP3GVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDENRWGY4TQNBUGM>.
You are receiving this because you commented.Message ID: ***@***.***>
|
感谢您的火速回复！！ |
|
您好作者,不好意思又来打扰!!! 我已经在H2中的mapping.py中的def run(self, first_frame, update_pose):函数加入您说的adaptive_kf_pruning，如下： ---------------------------------------------------------------------------------------------------------------------------------------------------------------> 并在最开始的class Mapping:中添加了对应的 -------------------------------------------------------------------------------------------------------------------------------------------------------------------> idx: 458 ^C******* mapping process died ******* 是我的使用方法不对吗T T |
|
和adaptive_kf_pruning有关的函数都要进行替换，比如keyframe_util.py中的内容。请仔细检查，或者直接使用H3-Mapping。
…________________________________
发件人: DeBinW ***@***.***>
发送时间: 2024年8月3日 23:51
收件人: SYSU-STAR/H2-Mapping ***@***.***>
抄送: JIANG Chenxing ***@***.***>; Comment ***@***.***>
主题: Re: [SYSU-STAR/H2-Mapping] 您好作者，我本次使用D455深度相机，跑您的开源代码H2-Mapping，有以下问题想和您请教一下 (Issue #45)
您好作者,不好意思又来打扰!!!
我已经在H2中的mapping.py中的def run(self, first_frame, update_pose):函数加入您说的adaptive_kf_pruning，如下：
------------------------------------------------------------------------------------------------------------------------------------------------------------------->
def run(self, first_frame, update_pose):
self.idx = 0
self.voxel_initialized = torch.zeros(self.num_vertexes).cuda().bool()
self.vertex_initialized = torch.zeros(self.num_vertexes).cuda().bool()
self.kf_unoptimized_voxels = None
self.kf_optimized_voxels = None
self.kf_all_voxels = None
if self.run_ros:
rospy.init_node('listener', anonymous=True)
# realsense
color_sub = message_filters.Subscriber(self.color_topic, Image)
depth_sub = message_filters.Subscriber(self.depth_topic, Image)
pose_sub = message_filters.Subscriber(self.pose_topic, Odometry)
ts = message_filters.ApproximateTimeSynchronizer([color_sub, depth_sub, pose_sub], 2, 1 / 10,
allow_headerless=False)
print(" ========== MAPPING START ===========")
ts.registerCallback(self.callback)
rospy.spin()
else:
if self.mesher is not None:
self.mesher.rays_d = first_frame.get_rays()
self.create_voxels(first_frame)
self.map_states, self.voxel_initialized, self.vertex_initialized = initemb_sdf(first_frame,
self.map_states,
self.sdf_truncation,
voxel_size=self.voxel_size,
voxel_initialized=self.voxel_initialized,
octant_idx=self.octant_idx,
vertex_initialized=self.vertex_initialized,
use_gt=self.use_gt)
self.sdf_priors = self.map_states["sdf_priors"]
self.insert_kf(first_frame)
self.do_mapping(tracked_frame=first_frame, update_pose=update_pose)
self.tracked_pose = first_frame.get_ref_pose().detach() @ first_frame.get_d_pose().detach()
ref_pose = self.current_kf.get_ref_pose().detach() @ self.current_kf.get_d_pose().detach()
rel_pose = torch.linalg.inv(ref_pose) @ self.tracked_pose
self.frame_poses += [(len(self.kf_graph) - 1, rel_pose.cpu())]
self.render_debug_images(first_frame)
print("mapping started!")
progress_bar = tqdm(range(self.start_frame, self.end_frame), position=0)
progress_bar.set_description("mapping frame")
for frame_id in progress_bar:
data_in = self.data_stream[frame_id]
if self.use_gt:
tracked_frame = RGBDFrame(*data_in[:-1], offset=self.offset, ref_pose=data_in[-1])
else:
tracked_frame = RGBDFrame(*data_in[:-1], offset=self.offset, ref_pose=self.tracked_pose.clone())
if update_pose is False:
tracked_frame.d_pose.requires_grad_(False)
if tracked_frame.ref_pose.isinf().any():
continue
self.mapping_step(frame_id, tracked_frame, update_pose)
print("******* mapping process died *******")
print(f"********** post-processing {self.final_iter} steps **********")
self.num_iterations = 1
for iter in range(self.final_iter):
self.do_mapping(tracked_frame=None, update_pose=False)
print("******* extracting final mesh *******")
**if not self.adaptive_kf_pruning:
pose = self.get_updated_poses()
else:
print("减小损耗！！！------------------------------")
pose = []
for ref_frame in self.kf_graph:
ref_pose = ref_frame.get_ref_pose().detach().cpu() @ ref_frame.get_d_pose().detach().cpu()
frame_pose = ref_pose
pose += [frame_pose.detach().cpu().numpy()]**
self.kf_graph = None
mesh = self.extract_mesh(res=self.mesh_res, clean_mesh=False, map_states=self.map_states)
self.logger.log_ckpt(self, name="final_ckpt.pth")
pose = np.asarray(pose)
pose[:, 0:3, 3] -= self.offset
self.logger.log_numpy_data(pose, "frame_poses")
self.logger.log_mesh(mesh)
self.logger.log_numpy_data(self.extract_voxels(map_states=self.map_states), "final_voxels")
print("******* mapping process died *******")
vis = o3d.visualization.Visualizer()
vis.create_window(window_name="results")
vis.get_render_option().point_size = 4
vis.get_render_option().mesh_show_back_face = False
vis.get_render_option().light_on = False
vis.add_geometry(mesh)
vis.run()
while True:
if not vis.poll_events():
break
vis.update_renderer()
vis.destroy_window()
-------------------------------------------------------------------------------------------------------------------------------------------------
并且在def insert_kf(self, frame):中添加了对应的代码如下：
----------------------------------------------------------------------------------------------------------------------------------------------->
def insert_kf(self, frame):
self.last_kf_observed = self.current_seen_voxel
self.current_kf = frame
self.last_kf_seen_voxel = self.seen_voxel
self.kf_graph += [frame]
self.kf_seen_voxel += [self.seen_voxel]
self.kf_seen_voxel_num += [self.last_kf_observed]
self.kf_svo_idx += [self.svo_idx]
**if self.adaptive_kf_pruning:
self.kf_insert_round += [self.round_num]
self.kf_latest_select_round += [0]**
# If a new keyframe is inserted,
# add the voxel in the newly inserted keyframe to the unoptimized voxel (remove the overlapping voxel)
if self.kf_selection_method == 'multiple_max_set_coverage' and self.kf_unoptimized_voxels != None:
self.kf_unoptimized_voxels[self.svo_idx.long() + 1] += True
self.kf_unoptimized_voxels[0] = False
________________________________
并在最开始的class Mapping:中添加了对应的
# used for adaptive keyframe pruning
self.round_num = 0
self.kf_latest_select_round = []
self.kf_insert_round = []
-------------------------------------------------------------------------------------------------------------------------------------------------------------------->
但是在运行过程中还是出现以下报错
idx: 458
idx: 459
idx: 460
idx: 461
idx: 462
idx: 463
idx: 464
idx: 465
[ERROR] [1722699436.076471]: bad callback: <bound method Subscriber.callback of <message_filters.Subscriber object at 0x7f89d351a100>>
Traceback (most recent call last):
File "/opt/ros/noetic/lib/python3/dist-packages/rospy/topics.py", line 750, in _invoke_callback
cb(msg)
File "/opt/ros/noetic/lib/python3/dist-packages/message_filters/init.py", line 76, in callback
self.signalMessage(msg)
File "/opt/ros/noetic/lib/python3/dist-packages/message_filters/init.py", line 58, in signalMessage
cb(*(msg + args))
File "/opt/ros/noetic/lib/python3/dist-packages/message_filters/init.py", line 330, in add
self.signalMessage(msgs)
File "/opt/ros/noetic/lib/python3/dist-packages/message_filters/init.py", line 58, in signalMessage
cb((msg + args))
File "/home/wdb/Packges/H2-Mapping/mapping/src/mapping.py", line 145, in callback
self.mapping_step(self.idx, tracked_frame, update_pose)
File "/home/wdb/Packges/H2-Mapping/mapping/src/mapping.py", line 165, in mapping_step
self.do_mapping(tracked_frame=tracked_frame, update_pose=update_pose)
File "/home/wdb/Packges/H2-Mapping/mapping/src/mapping.py", line 308, in do_mapping
bundle_adjust_frames(
File "/home/wdb/Packges/H2-Mapping/mapping/src/functions/render_helpers.py", line 526, in bundle_adjust_frames
scaler.scale(loss).backward()
File "/home/wdb/anaconda3/envs/h2mapping/lib/python3.8/site-packages/torch/_tensor.py", line 487, in backward
torch.autograd.backward(
File "/home/wdb/anaconda3/envs/h2mapping/lib/python3.8/site-packages/torch/autograd/init.py", line 197, in backward
Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 114.00 MiB (GPU 0; 7.73 GiB total capacity; 3.90 GiB already allocated; 90.50 MiB free; 4.23 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
^C******* mapping process died *******
********** post-processing 0 steps **********
******* extracting final mesh *******
减小损耗！！！------------------------------
********** get color from network **********
******* mapping process died *******
是我的使用方法不对吗T T
―
Reply to this email directly, view it on GitHub<#45 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/APMKKNTXLLC3EOB473L6L6LZPT4A5AVCNFSM6AAAAABLN7TP3GVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDENRWHA3DGMJVGI>.
You are receiving this because you commented.Message ID: ***@***.***>
|
可能是我，对应参数设置的问题？运行后实时建图效果根本看不出来是什么东西。
想跟您请教一下，看我更改配置的思路是否正确？
首先我进入到mapping/configs/realsense文件下更改了realsense.yaml的文件配置如下：
————————————————————————————————————————————————————————>
log_dir: './logs'
decoder: parallel_hash_net
update_pose: False
criteria:
rgb_weight: 0.5 # .5
depth_weight: 1
sdf_weight: 50000.0
fs_weight: 10.0
sdf_truncation: 0.05 # 0.1
decoder_specs:
voxel_size: 0.1 # same as mapper_specs
L: 4 # Number of levels
F_entry: 2 # Number of feature dimensions per entry
log2_T: 19 # each level's hashmap_size = F_entry * (2F_entry)
b: 2.0 # each level's resolution = N_min * (bLevel)
mapper_specs:
start_frame: 0
end_frame: -1
N_rays_each: 4096 # mapping's sampling ray
batch_size: 256
num_vertexes: 200000
inflate_margin_ratio: 0.1
voxel_size: 0.1
step_size: 0.1
num_iterations: 8
max_voxel_hit: 10
final_iter: 0
mesh_res: 8
overlap_th: 0.8
kf_window_size: 8
kf_selection_method: "multiple_max_set_coverage" # "random” or “multiple_max_set_coverage”
kf_selection_random_radio: 0.5 # random keyframe ratio
insert_method: "intersection" # "naive" or "intersection"
insert_ratio: 0.50
offset: 10 # used to make make the coordinate of each point positive
use_adaptive_ending: True # adaptive iteration
ros_args:
intrinsic: [ 382.653015136719, 381.758209228516, 319.552795410156, 249.586685180664 ] # K[0, 0], K[1, 1], K[0, 2], K[1, 2]
color_topic: '/camera/color/image_raw'
depth_topic: '/camera/aligned_depth_to_color/image_raw'
pose_topic: /vins_estimator/cam_pose
debug_args:
verbose: false
mesh_freq: -1
render_freq: -1
save_ckpt_freq: 100
render_res: [ 320, 240 ]
————————————————————————————————————————————————————————>
主要修改了：
1）
batch_size: 256
2）
insert_ratio: 0.50
3）
ros_args:
intrinsic: [ 382.653015136719, 381.758209228516, 319.552795410156, 249.586685180664 ] # K[0, 0], K[1, 1], K[0, 2], K[1, 2]
4）
save_ckpt_freq: 100
————————————————————————————————————————————————————————>
然后我进入到/home/wdb/Packages/H2-Mapping/src/dvins/config/uav2022文件下修改了uav_nerf.yaml文件，如下：
————————————————————————————————————————————————————————>
%YAML:1.0
#common parameters
imu_topic: "/camera/imu"
image_topic: "/camera/color/image_raw"
depth_topic: "/camera/aligned_depth_to_color/image_raw"
output_path: "/home/wdb/Dataset/"
#camera calibration
model_type: PINHOLE
camera_name: color
image_width: 640
image_height: 480
max_depth: 8.0 # max accurate depth of rgbd camera
min_depth: 0.1 # min accurate depth of rgbd camera
depth_cov: 0.005 # depth measurement cov when depth = 1
distortion_parameters:
k1: -0.0541978925466537
k2: 0.0667606219649315
p1: -0.000154328925418667
p2: -8.18331973277964e-05
projection_parameters:
fx: 382.653015136719
fy: 381.758209228516
cx: 319.552795410156
cy: 249.586685180664
Extrinsic parameter between IMU and Camera.
estimate_extrinsic: 0 # 0 Have an accurate extrinsic parameters. We will trust the following imu^R_cam, imu^T_cam, don't change it.
# 1 Have an initial guess about extrinsic parameters. We will optimize around your initial guess.
# 2 Don't know anything about extrinsic parameters. You don't need to give R,T. We will try to calibrate it. Do some rotation movement at beginning.
#If you choose 0 or 1, you should write down the following matrix.
#Rotation from camera frame to imu frame, imu^R_cam
extrinsicRotation: !!opencv-matrix
rows: 3
cols: 3
dt: d
data: [ 0.999997, 0.000193991, -0.00240292,
-0.000202728, 0.999993, -0.00363631,
0.0024022, 0.00363679, 0.999991]
#Translation from camera frame to imu frame, imu^T_cam
extrinsicTranslation: !!opencv-matrix
rows: 3
cols: 1
dt: d
data: [0.0290573518723249, 0.00749651249498129, 0.015767564997077]
#feature traker paprameters
max_cnt: 150 # max feature number in feature tracking
min_dist: 25 # min distance between two features
freq: 10 # frequence (Hz) of publish tracking result. At least 10Hz for good estimation. If set 0, the frequence will be same as raw image
F_threshold: 2.0 # ransac threshold (pixel)
show_track: 1 # publish tracking image as topic
equalize: 0 # if image is too dark or light, trun on equalize to find enough features
fisheye: 0 # if using fisheye, trun on it. A circle mask will be loaded to remove edge noisy points
use_fast: 1 # if it equals to 1, we extract and track fast feature, otherwise we use GFTT from opencv
use_rgbd: 1 # if using rgbd camera, turn on it. Depth residuals will be added to sliding window optimization and initialization
min_fast_resp: 5 # min response of fast feature
#optimization parameters
max_solver_time: 0.04 # max solver itration time (ms), to guarantee real time
max_num_iterations: 10 # max solver itrations, to guarantee real time
keyframe_parallax: 10.0 # keyframe selection threshold (pixel)
reproj_threshold: 5.0
#imu parameters The more accurate parameters you provide, the better performance
acc_n: 1.6464941447124525e-02 # accelerometer measurement noise standard deviation. #0.2 0.04
gyr_n: 1.8589544386089379e-03 # gyroscope measurement noise standard deviation. #0.05 0.004
acc_w: 7.7484923746061745e-04 # accelerometer bias random work noise standard deviation. #0.002
gyr_w: 2.2637222606574714e-05 # gyroscope bias random work noise standard deviation. #4.0e-5
g_norm: 9.805 # gravity magnitude
#loop_closure closure parameters
loop_closure: 0 # start loop_closure closure
fast_relocalization: 0 # useful in real-time and large project
load_previous_pose_graph: 0 # load and reuse previous pose graph; load from 'pose_graph_save_path'
pose_graph_save_path: "/home/wdb/Dataset/" # save and load path
#unsynchronization parameters
estimate_td: 0 # online estimate time offset between camera and imu
td: -0.015 # initial value of time offset. unit: s. readed image clock + td = real image clock (IMU clock)
#rolling shutter parameters
rolling_shutter: 0 # 0: global shutter camera, 1: rolling shutter camera
rolling_shutter_tr: 0.033 # unit: s. rolling shutter read out time per frame (from data sheet).
#visualization parameters
save_image: 1 # save image in pose graph for visualization prupose; you can close this function by setting 0
visualize_imu_forward: 0 # output imu forward propogation to achieve low latency and high frequence results
visualize_camera_size: 0.4 # size of camera marker in RVIZ
**————————————————————————————————————————————————————————>
其中主要去修改了IMU参数、相机内外参数，旋转平移矩阵
1）
#imu parameters The more accurate parameters you provide, the better performance
acc_n: 1.6464941447124525e-02 # accelerometer measurement noise standard deviation. #0.2 0.04
gyr_n: 1.8589544386089379e-03 # gyroscope measurement noise standard deviation. #0.05 0.004
acc_w: 7.7484923746061745e-04 # accelerometer bias random work noise standard deviation. #0.002
gyr_w: 2.2637222606574714e-05 # gyroscope bias random work noise standard deviation. #4.0e-5
g_norm: 9.805 # gravity magnitude
2）
distortion_parameters:
k1: -0.0541978925466537
k2: 0.0667606219649315
p1: -0.000154328925418667
p2: -8.18331973277964e-05
projection_parameters:
fx: 382.653015136719
fy: 381.758209228516
cx: 319.552795410156
cy: 249.586685180664
3）
extrinsicRotation: !!opencv-matrix
rows: 3
cols: 3
dt: d
data: [ 0.999997, 0.000193991, -0.00240292,
-0.000202728, 0.999993, -0.00363631,
0.0024022, 0.00363679, 0.999991]
#Translation from camera frame to imu frame, imu^T_cam
extrinsicTranslation: !!opencv-matrix
rows: 3
cols: 1
dt: d
data: [0.0290573518723249, 0.00749651249498129, 0.015767564997077]**
————————————————————————————————————————————————————————>
但是建图效果特别不好，并且运行bash ros_cmd/run_vins_rgbd.sh指令后会一直弹出
[ INFO] [1721887803.371261541]: IMU excitation not enouth!
正式运行TRACKING启动后就没有了
能麻烦您帮忙解答一下吗？或者提供相关思路，我自己再进行排查。
然后就是：
1）分辨率自行调整会影响到建图精度吗？比如我调到843*480
2）如果想提高精度的话从哪些方面入手呢？参数？设备？
------------------------------------------------------------------------感谢、感谢、感谢------------------------------------------------------------------
The text was updated successfully, but these errors were encountered: