Error during compilation: all CUDA-capable devices are busy or unavailable

[omar-khan]

Hi there,

During the install, I received plenty of messages complaining about:
/home/bakks/virginian/virginian/gtest/include/gtest/internal/gtest-port.h:1377: undefined reference to `operator new(unsigned int)'

Looking at your Makefiles, I noticed that your code has some kind of dependency on gsl-1.14 and gtest. So I performed a make gsl and make gtest which removed the error regarding the new operator. However, performing a make again now stalls with the following:

......... // not copied
a - ../release/sql/sql.tab.o
a - ../release/vm/virginia-single.o
a - ../release/vm/vm_gpu.o
cp virginian.h ../lib
/usr/bin/g++-4.4.7 -g3 -I../lib -Wall -I /usr/local/cuda/include -I . -I sql/ -c test/db.cc -o ../release/test/db.o
/usr/bin/g++-4.4.7 -g3 -I../lib -Wall -I /usr/local/cuda/include -I . -I sql/ -c test/growth.cc -o ../release/test/growth.o
/usr/bin/g++-4.4.7 -g3 -I../lib -Wall -I /usr/local/cuda/include -I . -I sql/ -c test/virgstate.cc -o ../release/test/virgstate.o
/usr/bin/g++-4.4.7 -g3 -I../lib -Wall -I /usr/local/cuda/include -I . -I sql/ -c test/timing.cc -o ../release/test/timing.o
/usr/bin/g++-4.4.7 -g3 -I../lib -Wall -I /usr/local/cuda/include -I . -I sql/ -c test/table.cc -o ../release/test/table.o
/usr/bin/g++-4.4.7 -g3 -I../lib -Wall -I /usr/local/cuda/include -I . -I sql/ -c test/sql.cc -o ../release/test/sql.o
/usr/bin/g++-4.4.7 -g3 -I../lib -Wall -I /usr/local/cuda/include -I . -I sql/ -c test/test.cc -o ../release/test/test.o
/usr/bin/g++-4.4.7 -lm -lpthread -L/usr/local/cuda/lib64 -lcuda -lcudart ../release/test/db.o ../release/test/growth.o ../release/test/virgstate.o ../release/test/timing.o ../release/test/table.o ../release/test/sql.o ../release/test/test.o ../lib/virginian.a ../lib/libgtest.a -lstdc++ -pthread -L../lib -lgsl -lgslcblas -o ../release/virginiantest
make -s -C ../db clean
make[2]: Entering directory '/home/tools/virginian/db'
make[2]: Leaving directory '/home/tools/virginian/db'
make -s -C ../db
make[2]: Entering directory '/home/tools/virginian/db'
make[2]: Leaving directory '/home/tools/virginian/db'
../db/generate ../db/comparedb 8000000
CUDA Allocating pinned tablet memory error: all CUDA-capable devices are busy or unavailable :: init.c line 59
0,Makefile:83: recipe for target '../db/comparedb' failed
make[1]: *** [../db/comparedb] Segmentation fault
make[1]: *** Deleting file '../db/comparedb'
make[1]: Leaving directory '/home/tools/virginian/src'
Makefile:4: recipe for target 'all' failed
make: *** [all] Error 2

Please note that I am able to run any CUDA or OpenCL program on my machine even after the above error. This happens even by running "../db/generate ../db/comparedb 8000000" manually which is not even a CUDA program !!!.

[bakks]

@omar-khan this project naively selects the cuda device using this variable:

virginian/src/virginian.h

Line 103 in 594d8b7

#define VIRG_CUDADEVICE 0

. I believe different integers correspond to different CUDA-capable devices, though I don't know how these are allocated. If you tweak that variable do you still see the same error?