Kokkos, ninja: make clean deletes generated cpp file and is not regenerated, causing compilation issues with rebuild when using ninja

[ndellingwood]

Bug Report

@trilinos/kokkos

Description

Following the merge of #13589 , some build system behavior changes, when using ninja as build generator, were reported by @jhux2 @cgcgcg .

When attempting to re-compile Trilinos after calling make clean, compilation fails due to a missing generated file Kokkos_Version_Info.cpp that was deleted during the make clean call, but not regenerated (no rerun of configure+generation stage) during a follow up build stage

• This only occurs with ninja as the build generator - I did not reproduce the issue with Unix Makefiles as generator
• This does not occur when following the same basic steps, but with the standalone kokkos repo (i.e. no Trilinos)

Steps to Reproduce

export TRILINOS_SRC=<path-to-your-source>

cmake -B build -G Ninja -DTrilinos_ENABLE_Kokkos=ON  -DCMAKE_CXX_COMPILER=g++ -DCMAKE_C_COMPILER=gcc ${TRILINOS_SRC}
cmake --build build
cmake --build build --target clean

# this next call triggers compilation error
cmake --build build

Error output:

[2/25] Building CXX object packages/kokkos/CMakeFiles/impl_git_version.dir/__/__/generated/Kokkos_Version_Info.cpp.o
FAILED: packages/kokkos/CMakeFiles/impl_git_version.dir/__/__/generated/Kokkos_Version_Info.cpp.o 
/opt/local/bin/g++  -I/Users/ndellin/Research/trilinos/Trilinos-kp/Build/Repr-post-release/build -I/Users/ndellin/Research/trilinos/Trilinos-kp/Build/Repr-post-release/build/generated -O3 -DNDEBUG -std=c++17 -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX14.5.sdk -MD -MT packages/kokkos/CMakeFiles/impl_git_version.dir/__/__/generated/Kokkos_Version_Info.cpp.o -MF packages/kokkos/CMakeFiles/impl_git_version.dir/__/__/generated/Kokkos_Version_Info.cpp.o.d -o packages/kokkos/CMakeFiles/impl_git_version.dir/__/__/generated/Kokkos_Version_Info.cpp.o -c /Users/ndellin/Research/trilinos/Trilinos-kp/Build/Repr-post-release/build/generated/Kokkos_Version_Info.cpp
cc1plus: fatal error: /Users/ndellin/Research/trilinos/Trilinos-kp/Build/Repr-post-release/build/generated/Kokkos_Version_Info.cpp: No such file or directory
compilation terminated.
[19/25] Building CXX object packages/kokkos/core/src/CMakeFiles/kokkoscore.dir/impl/Kokkos_Core.cpp.o
ninja: build stopped: subcommand failed.

Adding more knowledgeable folks than myself for possible insights or guidance on how to continue triage - @bartlettroscoe @masterleinad @dalg24 @crtrott

[masterleinad]

Looks similar to https://gitlab.kitware.com/cmake/cmake/-/issues/18032.

[bartlettroscoe]

Looks similar to gitlab.kitware.com/cmake/cmake/-/issues/18032.

From looking at the file kokkos/cmake/build_env_info.cmake, I don't think that is an identical problem. (In that case, there was an issue with not giving it an explicit absolute path.)

I don't remember all the details, but I you have to be careful about how you define custom targets/commands that generate files. It may be that:

Trilinos/packages/kokkos/cmake/build_env_info.cmake

Lines 103 to 106 in dc17e77

	add_custom_target(
	AlwaysCheckGit COMMAND ${CMAKE_COMMAND} -DRUN_CHECK_GIT_VERSION=1 -DKOKKOS_SOURCE_DIR=${Kokkos_SOURCE_DIR} -P
	${CURRENT_LIST_DIR}/build_env_info.cmake BYPRODUCTS ${post_configure_file}
	)

is the issue. I think you have to use a custom command, not a custom target (the documentation is confusing on this). See, for example:

https://github.com/TriBITSPub/TriBITS/blob/ade2bbbb57767f38b1e6efd464d50753717403fa/tribits/core/package_arch/TribitsCopyFilesToBinaryDir.cmake#L255-L259

Just a hutch (and a bad memory from 15+ years ago dealing with the same issue).

[masterleinad]

Looks like

diff --git a/packages/kokkos/cmake/build_env_info.cmake b/packages/kokkos/cmake/build_env_info.cmake
index ac28b2d8503..e9c6f8b8686 100644
--- a/packages/kokkos/cmake/build_env_info.cmake
+++ b/packages/kokkos/cmake/build_env_info.cmake
@@ -100,10 +100,11 @@ function(check_git_version)
 endfunction()
 
 function(check_git_setup)
-  add_custom_target(
-    AlwaysCheckGit COMMAND ${CMAKE_COMMAND} -DRUN_CHECK_GIT_VERSION=1 -DKOKKOS_SOURCE_DIR=${Kokkos_SOURCE_DIR} -P
-                           ${CURRENT_LIST_DIR}/build_env_info.cmake BYPRODUCTS ${post_configure_file}
+  add_custom_command(OUTPUT ${post_configure_file} COMMAND touch ${post_configure_file}
+    COMMAND ${CMAKE_COMMAND} -DRUN_CHECK_GIT_VERSION=1 -DKOKKOS_SOURCE_DIR=${Kokkos_SOURCE_DIR} -P
+                           ${CURRENT_LIST_DIR}/build_env_info.cmake
   )
+  add_custom_target(AlwaysCheckGit DEPENDS ${post_configure_file})
 
   add_library(impl_git_version ${CMAKE_BINARY_DIR}/generated/Kokkos_Version_Info.cpp)
   target_include_directories(impl_git_version PUBLIC ${CMAKE_BINARY_DIR}/generated)

would fix the issue.

[ndellingwood]

Thanks @masterleinad , I tried a similar change without success, didn't realize the COMMAND touch ${post_configure_file} addition was necessary 👍

When testing the rebuild after clean, I got this warning:

/Library/Developer/CommandLineTools/usr/bin/ranlib: file: packages/kokkos/libimpl_git_version.a(Kokkos_Version_Info.cpp.o) has no symbols
warning: /Library/Developer/CommandLineTools/usr/bin/ranlib: archive library: packages/kokkos/libimpl_git_version.a the table of contents is empty (no object file members in the library define global symbols)

Is that a concern?

More detailed summary of steps:

s1067110:Repr-post-release ndellin$ cmake --build build 
[25/25] Linking CXX static library packages/kokkos/containers/src/libkokkoscontainers.a
s1067110:Repr-post-release ndellin$ cmake --build build --target clean
[1/1] Cleaning all built files...
Cleaning... 25 files.
s1067110:Repr-post-release ndellin$ cmake --build build 
[6/25] Linking CXX static library packages/kokkos/libimpl_git_version.a
/Library/Developer/CommandLineTools/usr/bin/ranlib: file: packages/kokkos/libimpl_git_version.a(Kokkos_Version_Info.cpp.o) has no symbols
/Library/Developer/CommandLineTools/usr/bin/ranlib: file: packages/kokkos/libimpl_git_version.a(Kokkos_Version_Info.cpp.o) has no symbols
warning: /Library/Developer/CommandLineTools/usr/bin/ranlib: archive library: packages/kokkos/libimpl_git_version.a the table of contents is empty (no object file members in the library define global symbols)
[25/25] Linking CXX static library packages/kokkos/containers/src/libkokkoscontainers.a

[masterleinad]

It seems

diff --git a/packages/kokkos/cmake/build_env_info.cmake b/packages/kokkos/cmake/build_env_info.cmake
index ac28b2d8503..76afbb74b63 100644
--- a/packages/kokkos/cmake/build_env_info.cmake
+++ b/packages/kokkos/cmake/build_env_info.cmake
@@ -4,7 +4,7 @@ find_package(Git QUIET)
 
 set(CURRENT_LIST_DIR ${CMAKE_CURRENT_LIST_DIR})
 set(pre_configure_dir ${CMAKE_CURRENT_LIST_DIR})
-set(post_configure_dir ${CMAKE_BINARY_DIR}/generated)
+set(post_configure_dir ${CMAKE_CURRENT_BINARY_DIR}/generated)
 
 set(pre_configure_file ${pre_configure_dir}/Kokkos_Version_Info.cpp.in)
 set(post_configure_file ${post_configure_dir}/Kokkos_Version_Info.cpp)
@@ -105,7 +105,7 @@ function(check_git_setup)
                            ${CURRENT_LIST_DIR}/build_env_info.cmake BYPRODUCTS ${post_configure_file}
   )
 
-  add_library(impl_git_version ${CMAKE_BINARY_DIR}/generated/Kokkos_Version_Info.cpp)
+  add_library(impl_git_version ${CMAKE_CURRENT_BINARY_DIR}/generated/Kokkos_Version_Info.cpp)
   target_include_directories(impl_git_version PUBLIC ${CMAKE_BINARY_DIR}/generated)
   target_compile_features(impl_git_version PRIVATE cxx_raw_string_literals)
   add_dependencies(impl_git_version AlwaysCheckGit)

is the better fix.

[ndellingwood]

@masterleinad your newest suggestion worked well 👍

[masterleinad]

@ndellingwood Can you create a pull request?

[ndellingwood]

@ndellingwood Can you create a pull request?

@masterleinad yes, I'll put up a kokkos PR shortly along with patch match PR to Trilinos

[jhux2]

@ndellingwood I can confirm that building Trilinos withninja works again.

[jhux2]

and thanks for the quick fix!