GPU memory leak issue

[handyzeng]

Hi,
When I use GPU to do face detection, I found that the memory used of the GPU keep increasing. Maybe there is an memory leak issue.

The detector I used as flowing:
mtcnn_detector = MtcnnDetector(model_folder='model', ctx=mx.gpu(0), num_worker = 1, minsize = 40, accurate_landmark = True)

[YYuanAnyVision]

hmm, there are some related issues here:
apache/mxnet#2981
apache/mxnet#3055
looks like it's the problem with the memory manager(naive and pool allocation)...
there's nothing I can do here, but you can change to naivestorgemanager manually , like neodooth did.

[ZHAIXINGZHAIYUE]

我遇到了同样的问题
[16:46:26] /home/zhaixingzi/Applications/mxnet/dmlc-core/include/dmlc/./logging.h:235: [16:46:26] src/storage/./pooled_storage_manager.h:79: cudaMalloc failed: out of memory

[YYuanAnyVision]

--!# guess we have to wait for mxnet's solution.. otherwise you have to revert the memory manager

apache/mxnet#3055

any suggestion?

[YYuanAnyVision]

after change ptr = new storage::GPUPooledStorageManager(); to ptr = new storage::NaiveStorageManager<storage::GPUDeviceStorage>(); in src/storage/storage.cc line 76
the GPU memory stays in a range ..
but it will consume more memory and takes more time for memory allocation stuff.....

[ZHAIXINGZHAIYUE]

与原版matlab的代码相比，精度会不会下降？

[kli-casia]

The GPU memory keep increasing