CMake Build Failed: Proto/net.pb.h: No such file or directory

[haelyons]

Hello! Thanks for making this tool, I really want to use it to convert my games to the right format to use them in LC0!

I have made a Makefile from the project after installing all the dependencies (and already having LC0 installed), but have been unable to make the project due to the following error:

(base) haelyons@haelyons-VirtualBox:~/Downloads/trainingdata-tool/build$ make

[ 5%] Building CXX object CMakeFiles/trainingdata-tool.dir/src/PGNGame.cpp.o cc1plus: warning: /lc0/build/release/encoder_test.p/proto/net.pb.h: not a directory In file included from /lc0/src/neural/encoder.h:31:0, from /home/haelyons/Downloads/trainingdata-tool/src/PGNGame.h:4, from /home/haelyons/Downloads/trainingdata-tool/src/PGNGame.cpp:1:

/lc0/src/neural/network.h:33:10: fatal error: proto/net.pb.h: No such file or directory #include "proto/net.pb.h" ^~~~~~~~~~~~~~~~ compilation terminated.

CMakeFiles/trainingdata-tool.dir/build.make:62: recipe for target 'CMakeFiles/trainingdata-tool.dir/src/PGNGame.cpp.o' failed make[2]: *** [CMakeFiles/trainingdata-tool.dir/src/PGNGame.cpp.o] Error 1 CMakeFiles/Makefile2:67: recipe for target 'CMakeFiles/trainingdata-tool.dir/all' failed make[1]: *** [CMakeFiles/trainingdata-tool.dir/all] Error 2 Makefile:83: recipe for target 'all' failed make: *** [all] Error 2

The file proto/net.pb.h exists in the following locations within the lc0 directory, and I have tried adding them to my environment variable, and the CMakeLists.txt with little success:
build/release/encoder_test.p/proto
build/release/lc0.p/proto
build/release/liblc0_lib.so.p/proto`

I am running Ubuntu 18.04 LTS and have recloned and rebuilt lc0 to ensure that the correct versions of protobuf were installed and linked. This warning warning: /lc0/build/release/encoder_test.p/proto/net.pb.h: not a directory exists as I have tried adding proto/net.pb.h as an include directory, but it does not seem to be recognized.

I believe the issue is that the trainingdata-tool Cmake does not recognise the directory in which net.pb.h exists, but do not understand why this is the case (as I have added the correct address in the CMakeLists.txt) or how to fix it.

I hope this is enough information,
Thank you, H.

[haelyons]

Update: We're getting there, but slowly.

As suggested by Borg on the LC0 Help Channel (Discord) I ran the following:
git submodule update --init --recursive
git submodule update --remote

This fixed the error with the net.pb.h file in /proto, but revealed that more files were missing, specifically 'filesystem':
trainingdata-tool/src/TrainingDataReader.cpp:2:10: fatal error: filesystem: No such file or directory. Directly copying the file into trainingdata-tool/src/filesystem.h did not work, and so I am attempting to edit the reference to the header file, and the CMakeLists.txt

[haelyons]

Update: Error is still occurring, updating the compiler to use C17 and using relevant compiler flags to indicate this has not worked, in addition to the failed steps detailed in the previous entry.

So far, since the second update, I have tried:

• Updating g++ and gcc to be compatible with C17 (7.5.0)
• Updating to CMake 3.10.2
• Updating flags both within CMakeLists.txt and within gcc (global flags for libstdc++ which should contain (in C17) the desired filesystem.h
• Manually compiling the trainingdata-tool through g++
• Adding CXX flags to the current environment variable

[DanielUranga]

For GCC 7 I think you can try replacing #include <filesystem> with #include <experimental/filesystem>.

From https://stackoverflow.com/questions/45867379/why-does-gcc-not-seem-to-have-the-filesystem-standard-library

[haelyons]

I gave this a go but it resulted in quite a few syntax errors for filesystem calls, eventually updated to gcc 9.4.0 which got rid of the filesystem.h error, and progressed to the error below. It seems that the BMI2 headers / support is not found.

/usr/lib/gcc/x86_64-linux-gnu/9/include/bmi2intrin.h: In function ‘lczero::BitBoard lczero::{anonymous}::GetRookAttacks(lczero::BoardSquare, lczero::BitBoard)’: /usr/lib/gcc/x86_64-linux-gnu/9/include/bmi2intrin.h:76:1: error: inlining failed in call to always_inline ‘long long unsigned int _pext_u64(long long unsigned int, long long unsigned int)’: target specific option mismatch 76 | _pext_u64 (unsigned long long __X, unsigned long long __Y) | ^~~~~~~~~ /trainingdata-tool/lc0/src/chess/board.cc:387:29: note: called from here 387 | uint64_t index = _pext_u64(pieces.as_int(), rook_magic_params[square].mask_); | ~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ CMakeFiles/trainingdata-tool.dir/build.make:278: recipe for target 'CMakeFiles/trainingdata-tool.dir/lc0/src/chess/board.cc.o' failed

I am curious, was CMake or Ninja used for your builds? Also thanks for the response!

[haelyons]

After a bit more digging it seems that my machine does not have bmi support (either 1 or 2), verified through running the cat proc/cpuinfo | grep flags | uniq which returns:
fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc cpuid tsc_known_freq pni pclmulqdq monitor ssse3 cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single pti fsgsbase avx2 invpcid rdseed clflushopt md_clear flush_l1d

[DanielUranga]

I am curious, was CMake or Ninja used for your builds? Also thanks for the response!

It's CMake.

You can set the CPU architecture flags in order to avoid using PEXT. Check https://wiki.gentoo.org/wiki/GCC_optimization#-march.

[haelyons]

I am unfamiliar with this type of application (cpuid2cpuflags) - I have cloned the repository but cannot see a way to build the tool, and just running it as the README suggests is not functioning...

[haelyons]

Hello again, here with another update, I've been testing different compile flags for CMake, as it seems that the add_compile_definitions used was not yet implemented (or temporarily removed) for CMake 3.10.2. The formats that do not through errors are add_definitions(-DN0_PEXT) or add_compile_options(-DN0_PEXT=1). However, this still does not solve the error. Here is the error log from today after deleting /build and rebuilding:

root@haelyons-VirtualBox:/trainingdata-tool# rm -rf build && mkdir build && cd build && cmake .. && make -j
-- The C compiler identification is GNU 9.4.0
-- The CXX compiler identification is GNU 9.4.0
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Boost version: 1.65.1
-- Configuring done
-- Generating done
-- Build files have been written to: /trainingdata-tool/build
Scanning dependencies of target trainingdata-tool
[ 1%] Building CXX object CMakeFiles/trainingdata-tool.dir/src/TrainingDataReader.cpp.o
[ 53%] Building CXX object CMakeFiles/trainingdata-tool.dir/polyglot/src/game.cpp.o
[ 3%] Building CXX object CMakeFiles/trainingdata-tool.dir/src/PGNMoveInfo.cpp.o
[ 5%] Building CXX object CMakeFiles/trainingdata-tool.dir/src/polyglot_lib.cpp.o
[ 7%] Building CXX object CMakeFiles/trainingdata-tool.dir/lc0/src/neural/writer.cc.o
[ 8%] Building CXX object CMakeFiles/trainingdata-tool.dir/lc0/src/utils/random.cc.o
[ 10%] Building CXX object CMakeFiles/trainingdata-tool.dir/polyglot/src/fen.cpp.o
[ 12%] Building CXX object CMakeFiles/trainingdata-tool.dir/src/trainingdata.cpp.o
[ 80%] Building CXX object CMakeFiles/trainingdata-tool.dir/polyglot/src/move.cpp.o
[ 14%] Building CXX object CMakeFiles/trainingdata-tool.dir/polyglot/src/colour.cpp.o
[ 16%] Building CXX object CMakeFiles/trainingdata-tool.dir/polyglot/src/line.cpp.o
[ 17%] Building CXX object CMakeFiles/trainingdata-tool.dir/polyglot/src/io.cpp.o
[ 19%] Building CXX object CMakeFiles/trainingdata-tool.dir/src/TrainingDataDedup.cpp.o
[ 21%] Building CXX object CMakeFiles/trainingdata-tool.dir/polyglot/src/move_gen.cpp.o
[ 23%] Building CXX object CMakeFiles/trainingdata-tool.dir/polyglot/src/option.cpp.o
[ 83%] Building C object CMakeFiles/trainingdata-tool.dir/zlib/zutil.c.o
[ 25%] Building CXX object CMakeFiles/trainingdata-tool.dir/polyglot/src/pgn.cpp.o
[ 26%] Building CXX object CMakeFiles/trainingdata-tool.dir/src/TrainingDataWriter.cpp.o
[ 28%] Building CXX object CMakeFiles/trainingdata-tool.dir/polyglot/src/random.cpp.o
[ 30%] Building CXX object CMakeFiles/trainingdata-tool.dir/lc0/src/chess/bitboard.cc.o
[ 32%] Building CXX object CMakeFiles/trainingdata-tool.dir/polyglot/src/san.cpp.o
[ 33%] Building CXX object CMakeFiles/trainingdata-tool.dir/polyglot/src/square.cpp.o
[ 35%] Building CXX object CMakeFiles/trainingdata-tool.dir/polyglot/src/attack.cpp.o
[ 37%] Building CXX object CMakeFiles/trainingdata-tool.dir/src/trainingdata-tool.cpp.o
[ 39%] Building CXX object CMakeFiles/trainingdata-tool.dir/polyglot/src/util.cpp.o
[ 41%] Building CXX object CMakeFiles/trainingdata-tool.dir/lc0/src/utils/commandline.cc.o
[ 42%] Building CXX object CMakeFiles/trainingdata-tool.dir/polyglot/src/parse.cpp.o
[ 44%] Building C object CMakeFiles/trainingdata-tool.dir/zlib/crc32.c.o
[ 46%] Building CXX object CMakeFiles/trainingdata-tool.dir/lc0/src/chess/board.cc.o
[ 48%] Building CXX object CMakeFiles/trainingdata-tool.dir/lc0/src/utils/logging.cc.o
[ 50%] Building CXX object CMakeFiles/trainingdata-tool.dir/lc0/src/neural/encoder.cc.o
[ 51%] Building CXX object CMakeFiles/trainingdata-tool.dir/lc0/src/chess/position.cc.o
[ 55%] Building CXX object CMakeFiles/trainingdata-tool.dir/lc0/src/utils/filesystem.posix.cc.o
[ 57%] Building CXX object CMakeFiles/trainingdata-tool.dir/src/PGNGame.cpp.o
[ 58%] Building C object CMakeFiles/trainingdata-tool.dir/zlib/gzread.c.o
[ 60%] Building CXX object CMakeFiles/trainingdata-tool.dir/polyglot/src/board.cpp.o
[ 62%] Building C object CMakeFiles/trainingdata-tool.dir/zlib/infback.c.o
[ 64%] Building CXX object CMakeFiles/trainingdata-tool.dir/polyglot/src/hash.cpp.o
[ 66%] Building C object CMakeFiles/trainingdata-tool.dir/zlib/inflate.c.o
[ 67%] Building C object CMakeFiles/trainingdata-tool.dir/zlib/inftrees.c.o
[ 69%] Building CXX object CMakeFiles/trainingdata-tool.dir/polyglot/src/book.cpp.o
[ 71%] Building CXX object CMakeFiles/trainingdata-tool.dir/polyglot/src/list.cpp.o
[ 73%] Building C object CMakeFiles/trainingdata-tool.dir/zlib/adler32.c.o
[ 75%] Building CXX object CMakeFiles/trainingdata-tool.dir/polyglot/src/piece.cpp.o
[ 76%] Building CXX object CMakeFiles/trainingdata-tool.dir/polyglot/src/book_merge.cpp.o
[ 78%] Building CXX object CMakeFiles/trainingdata-tool.dir/polyglot/src/move_legal.cpp.o
[ 85%] Building C object CMakeFiles/trainingdata-tool.dir/zlib/compress.c.o
[ 87%] Building C object CMakeFiles/trainingdata-tool.dir/zlib/uncompr.c.o
[ 89%] Building C object CMakeFiles/trainingdata-tool.dir/zlib/trees.c.o
[ 91%] Building C object CMakeFiles/trainingdata-tool.dir/zlib/gzlib.c.o
[ 82%] Building CXX object CMakeFiles/trainingdata-tool.dir/polyglot/src/move_do.cpp.o
[ 92%] Building C object CMakeFiles/trainingdata-tool.dir/zlib/deflate.c.o
[ 94%] Building C object CMakeFiles/trainingdata-tool.dir/zlib/gzwrite.c.o
[ 96%] Building C object CMakeFiles/trainingdata-tool.dir/zlib/inffast.c.o
[ 98%] Building C object CMakeFiles/trainingdata-tool.dir/zlib/gzclose.c.o
/trainingdata-tool/zlib/gzwrite.c: In function ‘gz_comp’:
/trainingdata-tool/zlib/gzwrite.c:89:20: warning: implicit declaration of function ‘write’; did you mean ‘fwrite’? [-Wimplicit-function-declaration]
89 | writ = write(state->fd, strm->next_in, put);
| ^~~~~
| fwrite
/trainingdata-tool/zlib/gzwrite.c: In function ‘gzclose_w’:
/trainingdata-tool/zlib/gzread.c: In function ‘gz_load’:
/trainingdata-tool/zlib/gzread.c:35:15: warning: implicit declaration of function ‘read’; did you mean ‘fread’? [-Wimplicit-function-declaration]
35 | ret = read(state->fd, buf + have, get);
| ^~~~
| fread
/trainingdata-tool/zlib/gzread.c: In function ‘gzclose_r’:
/trainingdata-tool/zlib/gzlib.c: In function ‘gz_open’:
/trainingdata-tool/zlib/gzwrite.c:661:9: warning: implicit declaration of function ‘close’; did you mean ‘pclose’? [-Wimplicit-function-declaration]
661 | if (close(state->fd) == -1)
| ^~~~~
| pclose
/trainingdata-tool/zlib/gzlib.c:14:17: warning: implicit declaration of function ‘lseek’; did you mean ‘fseek’? [-Wimplicit-function-declaration]
14 | # define LSEEK lseek
| ^~~~~
/trainingdata-tool/zlib/gzlib.c:252:9: note: in expansion of macro ‘LSEEK’
252 | LSEEK(state->fd, 0, SEEK_END); / so gzoffset() is correct */
| ^~~~~
/trainingdata-tool/zlib/gzread.c:651:11: warning: implicit declaration of function ‘close’; did you mean pclose’? [-Wimplicit-function-declaration]
651 | ret = close(state->fd);
| ^~~~~
| pclose
In file included from /usr/lib/gcc/x86_64-linux-gnu/9/include/immintrin.h:105,
from /trainingdata-tool/lc0/src/chess/board.cc:39:
/usr/lib/gcc/x86_64-linux-gnu/9/include/bmi2intrin.h: In function ‘lczero::BitBoard lczero::{anonymous}::GetRookAttacks(lczero::BoardSquare, lczero::BitBoard)’:
/usr/lib/gcc/x86_64-linux-gnu/9/include/bmi2intrin.h:76:1: error: inlining failed in call to always_inline ‘long long unsigned int _pext_u64(long long unsigned int, long long unsigned int)’: target specific option mismatch
76 | _pext_u64 (unsigned long long __X, unsigned long long __Y)
| ^~~~~~~~~
/trainingdata-tool/lc0/src/chess/board.cc:387:29: note: called from here
387 | uint64_t index = pext_u64(pieces.as_int(), rook_magic_params[square].mask);
| ~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
CMakeFiles/trainingdata-tool.dir/build.make:278: recipe for target 'CMakeFiles/trainingdata-tool.dir/lc0/src/chess/board.cc.o' failed
make[2]: *** [CMakeFiles/trainingdata-tool.dir/lc0/src/chess/board.cc.o] Error 1
make[2]: *** Waiting for unfinished jobs....
CMakeFiles/Makefile2:67: recipe for target 'CMakeFiles/trainingdata-tool.dir/all' failed
make[1]: *** [CMakeFiles/trainingdata-tool.dir/all] Error 2
Makefile:83: recipe for target 'all' failed
make: *** [all] Error 2
root@haelyons-VirtualBox:/trainingdata-tool/build# exit
exit

Any help would be greatly appreciated.
Thanks, H.

[FrugoFruit90]

@haelyons did you manage to train an lc0 model on your games?
I've faced similar issues as you did, I also fixed the filesystem error by upgrading gcc to 9.4.0, now I a new issue, it seems like it's looking for a function that is not there? I downloaded lc0, polyglot and zlib libraries exactly as they were linked.
Or perhaps the issue is that all the above have to be built in order to work?

± |master U:1 ✗| → make -j
Consolidate compiler generated dependencies of target trainingdata-tool
[ 1%] Building CXX object CMakeFiles/trainingdata-tool.dir/src/TrainingDataWriter.cpp.o
/home/janek/Documents/trainingdata-tool/src/TrainingDataWriter.cpp: In member function ‘void TrainingDataWriter::WriteQueuedChunks(size_t)’:
/home/janek/Documents/trainingdata-tool/src/TrainingDataWriter.cpp:33:77: error: no matching function for call to ‘lczero::TrainingDataWriter::TrainingDataWriter(size_t&, std::__cxx11::basic_string)’
33 | dir_prefix + std::to_string(files_written / max_files_per_directory));
| ^
In file included from /home/janek/Documents/trainingdata-tool/src/TrainingDataWriter.h:12,
from /home/janek/Documents/trainingdata-tool/src/TrainingDataWriter.cpp:1:
/home/janek/Documents/trainingdata-tool/lc0/src/neural/writer.h:63:3: note: candidate: ‘lczero::TrainingDataWriter::TrainingDataWriter(int)’
63 | TrainingDataWriter(int game_id);
| ^~~~~~~~~~~~~~~~~~
/home/janek/Documents/trainingdata-tool/lc0/src/neural/writer.h:63:3: note: candidate expects 1 argument, 2 provided
/home/janek/Documents/trainingdata-tool/lc0/src/neural/writer.h:59:7: note: candidate: ‘lczero::TrainingDataWriter::TrainingDataWriter(const lczero::TrainingDataWriter&)’
59 | class TrainingDataWriter {
| ^~~~~~~~~~~~~~~~~~
/home/janek/Documents/trainingdata-tool/lc0/src/neural/writer.h:59:7: note: candidate expects 1 argument, 2 provided
CMakeFiles/trainingdata-tool.dir/build.make:131: recipe for target 'CMakeFiles/trainingdata-tool.dir/src/TrainingDataWriter.cpp.o' failed
make[2]: *** [CMakeFiles/trainingdata-tool.dir/src/TrainingDataWriter.cpp.o] Error 1
CMakeFiles/Makefile2:82: recipe for target 'CMakeFiles/trainingdata-tool.dir/all' failed
make[1]: *** [CMakeFiles/trainingdata-tool.dir/all] Error 2
Makefile:90: recipe for target 'all' failed
make: *** [all] Error 2

I will greatly appreciate any help that you might offer.
Best,
Jan