RuntimeError: CUDA error: an illegal memory access was encountered

[daviduarte]

Erro reproduction:

Download this video from CamNuvem robbery dataset:
Exec:
python3 scripts/demo.py video.source=354.mp4 video.output_dir=./output

It will raise an error:

CUDA error: an illegal memory access was encountered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

Traceback (most recent call last):
  File "/home/lecun/PHALP/phalp/trackers/PHALP.py", line 201, in track
    detections = self.get_human_features(image_frame, pred_masks, pred_bbox, pred_bbox_pad, pred_scores, frame_name, pred_classes, t_, measurments, gt_tids, gt_annots, extra_data)
  File "/home/lecun/PHALP/phalp/trackers/PHALP.py", line 410, in get_human_features
    hmar_out        = self.HMAR(masked_image_list.cuda(), **extra_args)
  File "/home/lecun/PHALP/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/lecun/PHALP/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/lecun/PHALP/scripts/camnuvem.py", line 119, in forward
    rend_depth = self.neural_renderer(pred_verts,
  File "/home/lecun/PHALP/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/lecun/PHALP/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/lecun/PHALP/venv/lib/python3.10/site-packages/neural_renderer/renderer.py", line 80, in forward
    return self.render_depth(vertices, faces, K, R, t, dist_coeffs, orig_size)
  File "/home/lecun/PHALP/venv/lib/python3.10/site-packages/neural_renderer/renderer.py", line 188, in render_depth
    images = nr.rasterize_depth(faces, self.image_size, self.anti_aliasing)
  File "/home/lecun/PHALP/venv/lib/python3.10/site-packages/neural_renderer/rasterize.py", line 461, in rasterize_depth
    return rasterize_rgbad(faces, None, image_size, anti_aliasing, near, far, eps, None, False, False, True)['depth']
  File "/home/lecun/PHALP/venv/lib/python3.10/site-packages/neural_renderer/rasterize.py", line 332, in rasterize_rgbad
    depth = depth[:, list(reversed(range(depth.shape[1]))), :]
RuntimeError: CUDA error: an illegal memory access was encountered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.