JITpackage not building on Windows

[DrTimothyAldenDavis]

When I try to build GraphBLAS in its own CMakeLists.txt build file (not as part of the root SuiteSparse build), I get this error. I'm using cmake 3.26 and am building the GraphBLAS project in Visual Studio:

1>------ Build started: Project: ZERO_CHECK, Configuration: Release x64 ------
1>Checking Build System
2>------ Build started: Project: grb_jitpackage, Configuration: Release x64 ------
2>Building Custom Rule C:/Users/davis/Documents/GitHub/GraphBLAS/JITpackage/CMakeLists.txt
2>grb_jitpackage.c
2>grb_jitpackage.vcxproj -> C:\Users\davis\Documents\GitHub\GraphBLAS\build\JITpackage\Release\grb_jitpackage.exe
3>------ Build started: Project: GB_JITpackage, Configuration: Release x64 ------
3>Generating compressed sources for JIT compiler...
3>C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\MSBuild\Microsoft\VC\v160\Microsoft.CppCommon.targets(231,5): error MSB6006: "cmd.exe" exited with code -1073740791.
3>Done building project "GB_JITpackage.vcxproj" -- FAILED.
4>------ Build started: Project: GraphBLAS, Configuration: Release x64 ------
4>Building Custom Rule C:/Users/davis/Documents/GitHub/GraphBLAS/CMakeLists.txt
4>GB_JITpackage.c
4>GB_prejit.c
...

I can't tell how the grb_jitpackage.exe is failing. I do see a grb_jitpackage.exe file that is compiled where I expect to see it (
Documents/GitHub/GraphBLAS/build/JITpackage/Release/grb_jitpackage.exe ).

My cmake configure output looks OK:

Selecting Windows SDK version 10.0.18362.0 to target Windows 10.0.19044.
The C compiler identification is MSVC 19.25.28611.0
Detecting C compiler ABI info
Detecting C compiler ABI info - done
Check for working C compiler: C:/Program Files (x86)/Microsoft Visual Studio/2019/Community/VC/Tools/MSVC/14.25.28610/bin/Hostx64/x64/cl.exe - skipped
Detecting C compile features
Detecting C compile features - done
Building SuiteSparse:GraphBLAS version: v9.3.0, date: June 1, 2024
GraphBLAS C API: v2.1, date: Dec 22, 2023
Source:           C:/Users/davis/Documents/GitHub/GraphBLAS 
Build:            C:/Users/davis/Documents/GitHub/GraphBLAS/build 
Install lib:      lib
Install include:  include/suitesparse
Install bin:      bin
Install pkg-file: lib
Install rpath:    $ORIGIN
Build   rpath:    
Build type:       Release 
Fortran:          not available
Looking for a CUDA compiler
Looking for a CUDA compiler - NOTFOUND
CUDA:             not found
CUDA:             not enabled
GraphBLAS CUDA JIT: disabled
GraphBLAS CPU JIT: enabled
Found OpenMP_C: -openmp (found version "2.0") 
Found OpenMP: TRUE (found version "2.0") found components: C 
GraphBLAS has OpenMP: ON
cpu_features (by google.com): enabled 
Performing Test GxB_HAVE_COMPLEX_C99
Performing Test GxB_HAVE_COMPLEX_C99 - Failed
Performing Test GxB_HAVE_COMPLEX_MSVC
Performing Test GxB_HAVE_COMPLEX_MSVC - Success
Building dynamic GraphBLAS library (no static)
Looking for fmax
Looking for fmax - found
C11 atomics: OK. -latomic not needed
CMAKE OpenMP libraries:    
CMAKE OpenMP include:      
CMAKE OpenMP C flags:      -openmp
CMAKE C flags: /DWIN32 /D_WINDOWS /O2 -wd"4244" -wd"4146" -wd"4018" -wd"4996" -wd"4047" -wd"4554" /O2 /Ob2 /DNDEBUG 
Skipping the demos in GraphBLAS/Demo
------------------------------------------------------------------------
JIT configuration:
------------------------------------------------------------------------
JIT C compiler: C:/Program Files (x86)/Microsoft Visual Studio/2019/Community/VC/Tools/MSVC/14.25.28610/bin/Hostx64/x64/cl.exe
JIT C flags:    /DWIN32 /D_WINDOWS /O2 -wd\"4244\" -wd\"4146\" -wd\"4018\" -wd\"4996\" -wd\"4047\" -wd\"4554\" /O2 /Ob2 /DNDEBUG -openmp
JIT link flags: /machine:x64
JIT lib prefix: 
JIT lib suffix: .dll
JIT obj suffix: .obj
JIT cache:      C:\Users\davis\AppData\Local/SuiteSparse/GrB9.3.0
JIT openmp inc: 
JIT openmp dirs 
JIT libraries:  
JIT cmake libs: 
------------------------------------------------------------------------
CMAKE report for: GraphBLAS
------------------------------------------------------------------------
inside common SuiteSparse root:  OFF
install in SuiteSparse/lib and SuiteSparse/include: OFF
build type:           Release
BUILD_SHARED_LIBS:    ON
BUILD_STATIC_LIBS:    OFF
C compiler:           C:/Program Files (x86)/Microsoft Visual Studio/2019/Community/VC/Tools/MSVC/14.25.28610/bin/Hostx64/x64/cl.exe 
C flags:              /DWIN32 /D_WINDOWS /O2 -wd"4244" -wd"4146" -wd"4018" -wd"4996" -wd"4047" -wd"4554" /O2 /Ob2 /DNDEBUG
C Flags release:      /O2 /Ob2 /DNDEBUG 
compile definitions:  _CRT_SECURE_NO_WARNINGS
------------------------------------------------------------------------
Configuring done (8.8s)
Generating done (0.2s)

In the SuiteSparse CI, the build looks different, and it succeeds, when using the SuiteSparse root CMakeLists.txt:

[1995/5428] Building C object GraphBLAS\JITpackage\CMakeFiles\grb_jitpackage.dir\Release\Source\grb_jitpackage.c.obj
[1996/5428] Linking C executable GraphBLAS\JITpackage\Release\grb_jitpackage.exe
[1997/5428] Generating compressed sources for JIT compiler...
Processing 242 input files ...

Total uncompressed: 1909807 bytes

Total compressed:   330640 bytes

Compression:        0.173127

[1998/5428] Building C object GraphBLAS\CMakeFiles\GraphBLAS.dir\Release\Source\aliased\GB_any_aliased.c.obj
[1999/5428] Building C object GraphBLAS\CMakeFiles\GraphBLAS.dir\Release\Config\GB_prejit.c.obj

@mmuetzel : any ideas?

@alugowski is seeing a similar problem here: GraphBLAS/python-suitesparse-graphblas#128 ,
see https://github.com/GraphBLAS/python-suitesparse-graphblas/actions/runs/9343575516/job/25713378326#step:9:351

[mmuetzel]

-1073740791 is 0xc0000409 in hexadecimal.
0xc0000409 means STATUS_STACK_BUFFER_OVERRUN.

I'm guessing that error is from running the JIT package generator executable (not actually from cmd.exe). That could be caused by an array index that is too large (positive or negative) when indexing a variable that is allocated on the stack. Maybe an erroneous integer typecast?
I wonder why that doesn't happen in the CI here though.

Which cmake command do you use to configure GRAPHBLAS?

[alugowski]

Is the problem visible in a Debug build? cmake -DCMAKE_BUILD_TYPE=Debug ....

I know MSVC's C++ Debug builds have bounds checks on iterators, maybe they also have checks that might help identify this. If not, maybe run in a debugger?

[mmuetzel]

Other differences to the CI are Visual Studio 2019 vs Visual Studio 2022 and Visual Studio vs. Ninja as the CMake generator.
But that shouldn't matter. Or could it?

[mmuetzel]

@alugowski: Unsure if this is related: UNIX Makefiles are sometimes a bit iffy on Windows. Do you have the option to switch to the Ninja generator in your CI rules? That is working better on Windows in my experience. (It is working fine on any platform where I tried it.)

[alugowski]

@alugowski: Unsure if this is related: UNIX Makefiles are sometimes a bit iffy on Windows. Do you have the option to switch to the Ninja generator in your CI rules? That is working better on Windows in my experience. (It is working fine on any platform where I tried it.)

Don't need to, Makefiles works in all cases we need. I remember I had to explicitly add that argument to solve an issue that I've forgotten, but I believe it was actually to fix something on Windows.

My preference is to use Ninja whenever possible as it's nicer on Unix too, but alas that wasn't an option. Note the official instructions only use make, no reference to the standard cmake commands. My preference is to stick to official instructions if possible, leave the defaults unchanged if possible, and only change settings if forced. Yes that Python repo is full of hacks, but each one was mandatory at some point and most probably still are.

[mmuetzel]

I tried to reproduce the error with a current head of the dev2 branch of the SuiteSparse repository (3df9cb323101e7a5b6b84ca9ee04a27e42e70068). Is that in sync with the current GraphBLAS repository?

I used the following commands (in .../SuiteSparse/GraphBLAS/build) using the "x64 Native Tools Command Prompt for VS 2022" (that is the "Community" edition) on Windows 11 23H2:

cmake ..
cmake --build . --config Release

That also picked the Visual Studio generator. But it is building without issues here. In particular, the JIT package generator was built and it generated the sources without issues. After that, building the object files continued without any errors or warnings.

Edit:
Result of the above configure step:

>cmake ..
-- Building for: Visual Studio 17 2022
-- Selecting Windows SDK version 10.0.22621.0 to target Windows 10.0.22631.
-- The C compiler identification is MSVC 19.40.33808.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: C:/Program Files/Microsoft Visual Studio/2022/Community/VC/Tools/MSVC/14.40.33807/bin/Hostx64/x64/cl.exe - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Building SuiteSparse:GraphBLAS version: v9.3.0, date: June 1, 2024
-- GraphBLAS C API: v2.1, date: Dec 22, 2023
-- Source:           D:/repo/SuiteSparse/SuiteSparse/GraphBLAS
-- Build:            D:/repo/SuiteSparse/SuiteSparse/GraphBLAS/build
-- Install lib:      lib
-- Install include:  include/suitesparse
-- Install bin:      bin
-- Install pkg-file: lib
-- Install rpath:    $ORIGIN
-- Build   rpath:
-- Build type:       Release
-- Looking for a Fortran compiler
-- Looking for a Fortran compiler - C:/Program Files (x86)/Intel/oneAPI/compiler/2024.1/bin/ifort.exe
-- The Fortran compiler identification is Intel 2021.0.0.20240222
-- Detecting Fortran compiler ABI info
-- Detecting Fortran compiler ABI info - done
-- Determine Intel Fortran Compiler Implicit Link Path
-- Determine Intel Fortran Compiler Implicit Link Path - done
-- Check for working Fortran compiler: C:/Program Files (x86)/Intel/oneAPI/compiler/2024.1/bin/ifort.exe - skipped
-- Fortran:          C:/Program Files (x86)/Intel/oneAPI/compiler/2024.1/bin/ifort.exe
-- Looking for a CUDA compiler
-- Looking for a CUDA compiler - NOTFOUND
-- CUDA:             not found
-- CUDA:             not enabled
-- GraphBLAS CUDA JIT: disabled
-- GraphBLAS CPU JIT: enabled
-- Found OpenMP_C: -openmp (found version "2.0")
-- Found OpenMP: TRUE (found version "2.0") found components: C
-- GraphBLAS has OpenMP: ON
-- cpu_features (by google.com): enabled
-- Performing Test GxB_HAVE_COMPLEX_C99
-- Performing Test GxB_HAVE_COMPLEX_C99 - Failed
-- Performing Test GxB_HAVE_COMPLEX_MSVC
-- Performing Test GxB_HAVE_COMPLEX_MSVC - Success
-- Building dynamic GraphBLAS library (no static)
-- Looking for fmax
-- Looking for fmax - found
-- Performing Test TEST_FOR_STDATOMIC
-- Performing Test TEST_FOR_STDATOMIC - Success
-- C11 atomics: OK. -latomic not needed
-- CMAKE OpenMP libraries:
-- CMAKE OpenMP include:
-- CMAKE OpenMP C flags:      -openmp
-- CMAKE C flags: /DWIN32 /D_WINDOWS /O2 -wd"4244" -wd"4146" -wd"4018" -wd"4996" -wd"4047" -wd"4554" /O2 /Ob2 /DNDEBUG
-- Skipping the demos in GraphBLAS/Demo
-- ------------------------------------------------------------------------
-- JIT configuration:
-- ------------------------------------------------------------------------
-- JIT C compiler: C:/Program Files/Microsoft Visual Studio/2022/Community/VC/Tools/MSVC/14.40.33807/bin/Hostx64/x64/cl.exe
-- JIT C flags:    /DWIN32 /D_WINDOWS /O2 -wd\"4244\" -wd\"4146\" -wd\"4018\" -wd\"4996\" -wd\"4047\" -wd\"4554\" /O2 /Ob2 /DNDEBUG -openmp
-- JIT link flags: /machine:x64
-- JIT lib prefix:
-- JIT lib suffix: .dll
-- JIT obj suffix: .obj
-- JIT cache:      C:\Users\Markus\AppData\Local/SuiteSparse/GrB9.3.0
-- JIT openmp inc:
-- JIT openmp dirs
-- JIT libraries:
-- JIT cmake libs:
-- ------------------------------------------------------------------------
-- CMAKE report for: GraphBLAS
-- ------------------------------------------------------------------------
-- inside common SuiteSparse root:  OFF
-- install in SuiteSparse/lib and SuiteSparse/include: OFF
-- build type:           Release
-- BUILD_SHARED_LIBS:    ON
-- BUILD_STATIC_LIBS:    OFF
-- C compiler:           C:/Program Files/Microsoft Visual Studio/2022/Community/VC/Tools/MSVC/14.40.33807/bin/Hostx64/x64/cl.exe
-- C flags:              /DWIN32 /D_WINDOWS /O2 -wd"4244" -wd"4146" -wd"4018" -wd"4996" -wd"4047" -wd"4554" /O2 /Ob2 /DNDEBUG
-- C Flags release:      /O2 /Ob2 /DNDEBUG
-- compile definitions:  _CRT_SECURE_NO_WARNINGS
-- ------------------------------------------------------------------------
-- Configuring done (83.9s)
-- Generating done (0.2s)
-- Build files have been written to: D:/repo/SuiteSparse/SuiteSparse/GraphBLAS/build

Relevant part of the build step:

>cmake --build . --config Release
MSBuild-Version 17.10.4+10fbfbf2e für .NET Framework

  1>Checking Build System
  Building Custom Rule D:/repo/SuiteSparse/SuiteSparse/GraphBLAS/JITpackage/CMakeLists.txt
  grb_jitpackage.c
  grb_jitpackage.vcxproj -> D:\repo\SuiteSparse\SuiteSparse\GraphBLAS\build\JITpackage\Release\grb_jitpackage.exe
  Generating compressed sources for JIT compiler...
  Processing 242 input files ...
  Total uncompressed: 1909807 bytes
  Total compressed:   330640 bytes
  Compression:        0.173127
  Building Custom Rule D:/repo/SuiteSparse/SuiteSparse/GraphBLAS/JITpackage/CMakeLists.txt
  Building Custom Rule D:/repo/SuiteSparse/SuiteSparse/GraphBLAS/CMakeLists.txt
  GB_JITpackage.c
  GB_prejit.c
  GB_all_aliased.c
  GB_any_aliased.c
  GB_is_shallow.c
  GB_apply.c
[...]

[mmuetzel]

Fwiw, the generator executable only links to the C runtime and the kernel for me:

$ ntldd grb_jitpackage.exe
        VCRUNTIME140.dll => C:\WINDOWS\SYSTEM32\VCRUNTIME140.dll (0x0000015518ad0000)
        KERNEL32.dll => C:\WINDOWS\SYSTEM32\KERNEL32.dll (0x0000015518af0000)

Or with Visual Studio tools:

...\SuiteSparse\GraphBLAS\build>dumpbin /dependents .\JITpackage\Release\grb_jitpackage.exe
Microsoft (R) COFF/PE Dumper Version 14.40.33808.0
Copyright (C) Microsoft Corporation.  All rights reserved.


Dump of file .\JITpackage\Release\grb_jitpackage.exe

File Type: EXECUTABLE IMAGE

  Image has the following dependencies:

    VCRUNTIME140.dll
    api-ms-win-crt-stdio-l1-1-0.dll
    api-ms-win-crt-heap-l1-1-0.dll
    api-ms-win-crt-runtime-l1-1-0.dll
    api-ms-win-crt-math-l1-1-0.dll
    api-ms-win-crt-locale-l1-1-0.dll
    KERNEL32.dll

  Summary

        1000 .data
        2000 .pdata
        5000 .rdata
        1000 .reloc
        1000 .rsrc
       50000 .text

[mmuetzel]

Just another idea: The maximum number of characters for a command line in cmd.exe is 8191 (in some Windows versions):
https://learn.microsoft.com/en-us/troubleshoot/windows-client/shell-experience/command-line-string-limitation

The sources are checked out at a deeper absolute path for you than they are for me.
A glob command is used to generate a string that contains the absolute paths to all relevant sources files.
That string is passed as an argument list to the JIT package generator.

It might be that the longer base path on your checkout location leads to the command line length limit being exceeded.

Does it work if you checkout the GraphBLAS sources to a path that is shorter (e.g., to C:\repo\GraphBLAS)?
If it does, a potential solution could be to write the list of relevant source files to a text file and pass that text file as an argument to the JIT package generator. The JIT package generator would need to be changed to read the file list from that text file instead of iterating through its command line arguments.

[mmuetzel]

I opened #302 to address the potential issue described in the previous comment.

[DrTimothyAldenDavis]

Just another idea: The maximum number of characters for a command line in cmd.exe is 8191:
https://learn.microsoft.com/en-us/troubleshoot/windows-client/shell-experience/command-line-string-limitation

Oh -- that's got to be it. There are 243 or so files in the JITpackage, and I'm working in a folder that is C: ... something here ... Documents/Github/GraphBLAS/build ...

Leave it MS to report error MSB6006: "cmd.exe" exited with code -1073740791 and not "error: command line too long ...". Sigh.

The 'something here' is my Windows home folder (I'm not on my Windows machine at the moment). I think that itself might be deeper than normal because it's a departmentally-managed machine. I don't have admin access, so I can't do C:/my/GraphBLAS. Adding to the complexity is that I so rarely use Windows at all, for anything, let alone development.

[DrTimothyAldenDavis]

I will try to shorten the paths of my files to see if that works. I will rewrite the JITpackage to write out the set of filenames to a file, in the cmake build process.

[DrTimothyAldenDavis]

Yes, the dev2 branch of GraphBLAS and the dev2 branch of SuiteSparse are currently identical. I also tend to keep the default branches of the 2 repos synced.

When I started GraphBLAS, I didn't place it inside my SuiteSparse repo because it was a stand-alone project, but then I decided later it should be part of SuiteSparse. But by then, lots of people were getting GraphBLAS from its own repo, not inside SuiteSparse. Thus the two repos. I guess eventually I should close down the stand-alone GraphBLAS repo and use SuiteSparse as a whole, instead, since it's more comprehensive (building GraphBLAS from the root CMakeLists.txt in SuiteSparse is very handy on Windows). Plus the github CI is far better.

[DrTimothyAldenDavis]

I will rewrite the JITpackage to write out the set of filenames to a file, in the cmake build process.

Oh you beat me to it :-) ... #302 ... thanks!

[DrTimothyAldenDavis]

Fixed with #302 and subsequent updates.