CUDA out of memory issue #10
Comments
|
I'm facing the same error. Titan X, 11.92GB memory available. |
|
Hi @danieleghisi @ianni67 I have definitely run the demo with less memory. 6 GB I think. Did you have any luck? |
|
Not really, still same issue for me...
…----- Messaggio originale -----
Da: "Tom Le Paine" <[email protected]>
Inviato: 01/02/2017 01:53
A: "tomlepaine/fast-wavenet" <[email protected]>
Cc: "danieleghisi" <[email protected]>; "Mention" <[email protected]>
Oggetto: Re: [tomlepaine/fast-wavenet] CUDA out of memory issue (#10)
Hi @danieleghisi @ianni67 I have definitely run the demo with less memory. 6 GB I think. Did you have any luck?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.
|
|
I'm very sorry for my late reply. I eventually solved the issue, and, alas, (or luckily?) it was my own fault. I did not manage the GPUs correctly, so two processes where contending the same GPU, both requesting all its memory. Confining the processes on separate GPUs (by using CUDA_VISIBLE_DEVICES appropriately) solved the problem. |
|
@ianni67, I'm glad you resolved your issue. @danieleghisi do you think you have the same problem? |
|
Sorry for the late reply. CUDA_VISIBLE_DEVICES solves the memory issues, but now I get a CuDNN version error:
Looks like an issue in tensorflow installation (I can't update CuDNN, due to other dependencies.) Daniele |
|
@tomlepaine Thanks Tom, I've managed to update CuDNN and dependencies, and the training works fine! On the other hand, the generation is always a constant value, the seed (perfectly flat waveform)... May I be doing something wrong? Thanks again for your support, |
|
@danieleghisi why are you saving predictions[0]? What is the shape of predictions? Glad you are closer to getting the code working! |
|
Hi Tom, |
|
@danieleghisi if you have succeeded in training it now, can you please share the model. |
|
Setting "gpu_fraction = .95" instead of 1 in the demo.py solve the problem for me! |
|
@francois-baptiste can you share the model. |
Hi, I'm having an "out of memory" issue while running the demo.
Snippet:
I tensorflow/core/common_runtime/gpu/gpu_device.cc:885] Found device 0 with properties:
name: GeForce GTX TITAN X
major: 5 minor: 2 memoryClockRate (GHz) 1.076
pciBusID 0000:04:00.0
Total memory: 11.91GiB
Free memory: 11.67GiB
(full log below)
I have tried to lower the model parameters, but nothing seems to work. Do you have any advice?
Why does the demo take so much GPU memory?
Thanks a lot,
Daniele
Full log:
python demo.py
I tensorflow/stream_executor/dso_loader.cc:128] successfully opened CUDA library libcublas.so locally
I tensorflow/stream_executor/dso_loader.cc:119] Couldn't open CUDA library libcudnn.so. LD_LIBRARY_PATH:
I tensorflow/stream_executor/cuda/cuda_dnn.cc:3459] Unable to load cuDNN DSO
I tensorflow/stream_executor/dso_loader.cc:128] successfully opened CUDA library libcufft.so locally
I tensorflow/stream_executor/dso_loader.cc:128] successfully opened CUDA library libcuda.so.1 locally
I tensorflow/stream_executor/dso_loader.cc:128] successfully opened CUDA library libcurand.so locally
I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
I tensorflow/core/common_runtime/gpu/gpu_device.cc:885] Found device 0 with properties:
name: GeForce GTX TITAN X
major: 5 minor: 2 memoryClockRate (GHz) 1.076
pciBusID 0000:04:00.0
Total memory: 11.91GiB
Free memory: 11.67GiB
W tensorflow/stream_executor/cuda/cuda_driver.cc:590] creating context when one is currently active; existing: 0x48c4140
I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
I tensorflow/core/common_runtime/gpu/gpu_device.cc:885] Found device 1 with properties:
name: GeForce GTX 980
major: 5 minor: 2 memoryClockRate (GHz) 1.342
pciBusID 0000:0a:00.0
Total memory: 3.94GiB
Free memory: 487.88MiB
W tensorflow/stream_executor/cuda/cuda_driver.cc:590] creating context when one is currently active; existing: 0x48c0320
E tensorflow/core/common_runtime/direct_session.cc:135] Internal: failed initializing StreamExecutor for CUDA device ordinal 2: Internal: failed call to cuDevicePrimaryCtxRetain: CUDA_ERROR_OUT_OF_MEMORY; total memory reported: 18446744073648275456
Traceback (most recent call last):
File "demo.py", line 16, in
gpu_fraction=gpu_fraction)
File "/home/daniele/fast-wavenet-master/wavenet/models.py", line 54, in init
sess = tf.Session(config=tf.ConfigProto(gpu_options=gpu_options))
File "/home/daniele/.local/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1186, in init
super(Session, self).init(target, graph, config=config)
File "/home/daniele/.local/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 551, in init
self._session = tf_session.TF_NewDeprecatedSession(opts, status)
File "/usr/lib/python2.7/contextlib.py", line 24, in exit
self.gen.next()
File "/home/daniele/.local/lib/python2.7/site-packages/tensorflow/python/framework/errors_impl.py", line 469, in raise_exception_on_not_ok_status
pywrap_tensorflow.TF_GetCode(status))
tensorflow.python.framework.errors_impl.InternalError: Failed to create session.
The text was updated successfully, but these errors were encountered: