Spec-2017

[Sezoir]

Enable building and execution of the SPEC 2017 benchmarks with the LLVM Flang compiler.

[Sezoir]

Next steps: I am not planning to take on this project. This is just an initial report found. Please feel free to work on this.

SPEC 2017:

Since SPEC 2017 consists of quite a lot of benchmarks, for this initial report I looked at exchange2 and bwaves_r benchmarks.

Initial report for exchange2:

• Minimum version of Fortran required using gfortran: F95
• How to compile with gfortran:
  The exchange2 benchmark consists of a single Fortran file.
  1. First cd to the benchmark with:

  cd benchspec/CPU/548.exchange2_r/src
  

  2. Now you can compile the project by running:

  gfortran exchange2.F90
  

• Does it compile with flang-new: Yes
  Similarly, to compiling with gfortran, you can compile with flang-new with:

  flang-new exchange2.F90
  

Initial report for bwaves_r:

• Minimum version of Fortran required using gfortran: F95
• How to compile with gfortran:
  The exchange2 benchmark consists of a single Fortran file.
  1. First cd to the benchmark with:

  cd benchspec/CPU/503.bwaves_r/src
  

  2. Now you can compile the project by running:

  gfortran -DSPEC -DSPEC_SUPPRESS_OPENMP block_solver.F flow_lam.F flux_lam.F jacobian_lam.F shell_lam.F fill1.F fill2.F
  

  If you want to compile with OpenMP then replace "-DSPEC_SUPPRESS_OPENMP" from above with "-D_OPENMP -fopenmp".
• Does it compile with flang-new: Yes
  To compile with flang-new you must have a build of PGMATH. I use classic flang to get a build of PGMATH. Then you can compile with flang-new with:

  flang-new -w -DSPEC -DSPEC_SUPPRESS_OPENMP -L/path/to/flang/runtime/build/lib -lpgmath block_solver.F flow_lam.F flux_lam.F jacobian_lam.F shell_lam.F fill1.F fill2.F
  

  Where the path to your PGMATH build lib directory is correctly modified.
  Again, like gfortran, if you want to compile with OpenMP then replace -DSPEC_SUPPRESS_OPENMP from above with -D_OPENMP -fopenmp. However, I found that when compiling with OpenMP, I end up getting a stack dump. I think this may be a lowering issue, but I am not sure. A small reproducer of code based of bwaves_r causing the error is:

  integer :: i,j,k,l
  real :: r2 = 0.0
  !$OMP PARALLEL DO PRIVATE(l,i,j,k) REDUCTION(+:r2) 
  do l=1,nb
      r2 =r(l,i,j,k)**2
  enddo
  !$OMP END PARALLEL DO
  end
  

Versions:

• gfortran version used: 9.3.0
• flang-new, fir-dev branch, HEAD: fb8eb63
• pgmath built from classic flang, HEAD: e7dedbf9200e740bf38b37892597498fd7dcbb41
• Architecture for compilation: aarch64

[rovka]

I did a few runs of SPEC on AArch64 using the scripting infrastructure from the llvm-test-suite (commit d52a93a8, plus a minor hack to remove -Werror=date-time from CMakeLists.txt) and a patched up cmake.

For reference, gfortran can handle all of SPEC with "-cpp -ffree-line-length-none -ffixed-line-length-none -Wno-argument-mismatch". With an additional "-std=f2003", it can handle everything except 521/621.wrf, and with "-std=f95" it can only handle 503/603.bwaves and 554/654.roms (the others fail to compile because they use various f2003 features such as command_argument_count, the ISO_C_BINDING module etc).

With flang built from fir-dev at fa74e15 and libpgmath at 7566954d, I'm getting the following results:

• 548/648.exchange_2 builds & runs successfully 🥳
• 503/603.bwaves, 549/649.fotonik3d and 554/654.roms fail at runtime
• 507/607.cactuBSSN, 527/627.cam4, 628.pop2 and 521/621.wrf fail to compile

I'm attaching the full build & run logs.
build-spec-flang.log
run-spec-flang.log

I intend to look into bwaves and roms first, since gfortran can deal with them in f95 mode.

[kiranchandramohan]

@rovka Thanks for the report. Shall we create separate entries for bwaves and roms? We might look into what is missing in others. And will create separate entries if we start working on any of the others.

[rovka]

@kiranchandramohan I think separate entries are probably a good idea. I'm considering this on hold until we're done with upstreaming though, since there's no point in exercising the fir-dev branch if it's frozen, and I'm not sure everything we need is in main yet.

[banach-space]

@kiranchandramohan I think separate entries are probably a good idea.

+1 :)

I'm considering this on hold

We are actually poking around a bit :) If we have something new to share, we will create dedicated tickets. From my initial triaging, the following split would make sense:

• 548/648.exchange_2
• 503/603.bwaves
• 549/649.fotonik3d
• 554/654.roms
• 507/607.cactuBSSN
• 527/627.cam4
• 628.pop2
• 521/621.wrf

What do you think? This would match how the benchmarks are split within SPEC. Also, for me each benchmark in a given pair seems to fail for the same reason (again, .

On a slightly different note @rovka, I was able to reproduce your results by running SPEC directly (i.e. outside LLVM's test suite). I have a couple of questions.

Q1
I used a modified version of "Example-gcc-linux-aarch64.cfg" from SPEC. Is that what you used?

There's an LLVM config file too, but that one led to quite a few linker issues. I've not tried fixing that yet.

Q2
I had to patch flang-new to include -lpgmath and -lpthread in the linker invocation. I couldn't find a way to work around that in the SPEC config file, though I'm fairly confident there's one. Did you you also need a workaround? Perhaps LLVM's test suite sorts that out for you?

Thanks!
-Andrzej

[rovka]

@banach-space I haven't looked into how the test suite sets this up. I didn't notice anything related to -lpthread, but I did pass flags for lpgmath. Basically I configured the test-suite with -DCMAKE_Fortran_FLAGS="-lpgmath -L$HOME/work/pgmath/build/lib -flang-experimental-exec" and it was all taken care of :)

[rovka]

This is now tracked in the LLVM organization.

[muke101]

With flang built from fir-dev at fa74e15 and libpgmath at 7566954d, I'm getting the following results:

* 548/648.exchange_2 builds & runs successfully partying_face

* 503/603.bwaves, 549/649.fotonik3d and 554/654.roms fail at runtime

* 507/607.cactuBSSN, 527/627.cam4, 628.pop2 and 521/621.wrf fail to compile

Hi all, is there an update on the status of this anywhere? Looking into compiling spec2017 with flang now and trying to figure out what can and can't be run. I've looked at the collection of issues tracked on the llvm repo but it's difficult to ascertain from those the updated status of the list above. Thanks.

[kiranchandramohan]

The rate version (given below) should pass at O3 on both AArch64 and X86_64. There are some issues with mcpu=native and Ofast.
503.bwaves_r 507.cactuBSSN_r 521.wrf_r 527.cam4_r 548.exchange2_r 549.fotonik3d_r 554.roms_r

For speed, I have not been tracking regularly. But the following used to pass at O3:
603.bwaves_s 607.cactuBSSN_s 621.wrf_s 648.exchange2_s 654.roms_s
And the following using to fail at O3:
627.cam4_s 628.pop2_s 649.fotonik3d_s

[KanRobert]

Do all the speed workloads pass now?

[kiranchandramohan]

Do all the speed workloads pass now?

They all began passing sometime late last year. But I have not been tracking this regularly.

[KanRobert]

Do all the speed workloads pass now?

They all began passing sometime late last year. But I have not been tracking this regularly.

Where can I find the config file for flang?

[KanRobert]

Do all the speed workloads pass now?

They all began passing sometime late last year. But I have not been tracking this regularly.

Where can I find the config file for flang?

Never mind. I asked this b/c I encountered some run error. Now, most of them are resolved by

1. Add llvm-project/build/lib to the env LD_LIBRARY_PATH
2. Add -fconvert=big-endian to FPORTABILITY for wrf and pop2
3. Set ulimit -s unlimited; ulimit -n 2000 before running

[KanRobert]

FYI, all speed benchmarks passed with -march=skylake-avx512 -g -O2/-O3 -flto for x86 with HEAD llvm@ba60d8a

[edisonchan]

performance of 521/621 is only half of gcc 14.1
llvm 19.1.0 (clang,clang++,flang-new) -march=x86-64 -Ofast

gcc 14.1 (gcc, g++, gfortran) -march=x86-64 -Ofast