Add support of GFortran in MKL package #5258

jbloino · 2024-09-17T12:53:29Z

Add support for Fortran/GFortran with MKL

This patch addresses #5254 and introduces 3 modifications to the package MKL:

added support of Fortran and GFortran to compile with the MKL libraries
changed download paths following some changes made on the Intel channel of conda (see https://community.intel.com/t5/Intel-Integrated-Performance/Problems-installing-with-conda-HTTP-403-FORBIDDEN/td-p/1611876 for details).
added support of latest version of MKL as of September 2024: 2024.2.1

Note that the support of Intel processors on OS X seems to have been dropped by Intel. No path could be found.

Content of the repository can be parsed from the JSON files in each architecture-dependent path:
https://software.repos.intel.com/python/conda//repodata.json

with being one of the following:

"linux-32"
"linux-64"
"win-32"
"win-64"

Minimum example with GFortran:

Fortran code

program test_mkl

    use iso_fortran_env, only: real64

    integer, parameter :: N = 10
    integer :: info
    integer, dimension(N) :: iwork
    real(real64), dimension(N,N) :: A, B, C

    call random_number(A)
    call random_number(B)

    call dgemm('T','N',N,N,N,1.0_real64,A,N,B,N,0.0_real64,C,N)

    call dgetrf(N,N,C,N,iwork,info)

end program test_mkl

xmake.lua

add_rules("mode.debug", "mode.release")

add_repositories("local-repo xmake-repo")
add_requires("mkl", { configs = { threading = "gomp" } })

target("test_mkl")
    set_kind("binary")
    set_policy("check.auto_ignore_flags", false)
    add_packages("mkl")
    add_files("test_mkl.f90")
    add_tests("default")

This patch introduces 3 modifications to the package MKL: - added support of Fortran and GFortran to compile with the MKL libraries - changed download paths following some changes made on the Intel channel of conda (see https://community.intel.com/t5/Intel-Integrated-Performance/Problems-installing-with-conda-HTTP-403-FORBIDDEN/m-p/1631577 for details). - added support of latest version of MKL as of September 2024: 2024.2.1 Note that the support of Intel processors on OS X seems to have been dropped by Intel. No path could be found. Content of the repository can be parsed from the JSON files in each architecture-dependent path: https://software.repos.intel.com/python/conda/<arch>/repodata.json with <arch> being one of the following: - "linux-32" - "linux-64" - "win-32" - "win-64" Minimum example with GFortran: Fortran code ``` program test_mkl use iso_fortran_env, only: real64 integer, parameter :: N = 10 integer :: info integer, dimension(N) :: iwork real(real64), dimension(N,N) :: A, B, C call random_number(A) call random_number(B) call dgemm('T','N',N,N,N,1.0_real64,A,N,B,N,0.0_real64,C,N) call dgetrf(N,N,C,N,iwork,info) end program test_mkl ``` xmake.lua ``` add_rules("mode.debug", "mode.release") add_repositories("local-repo xmake-repo") add_requires("mkl", { configs = { threading = "gomp" } }) target("test_mkl") set_kind("binary") set_policy("check.auto_ignore_flags", false) add_packages("mkl") add_files("test_mkl.f90") add_tests("default") ```

xq114 · 2024-09-18T03:35:55Z

https://software.repos.intel.com/python/conda/osx-64/repodata.json

The repository was also available in the new repository. Versions listed in the package and not available in the repository have been removed or replaced with more recent ones from the same year.

jbloino · 2024-09-18T08:13:41Z

https://software.repos.intel.com/python/conda/osx-64/repodata.json

Thank you very much. I have updated the package, restoring support of Intel Mac OS X and correcting the version.

Fixed URLs to some Windows 32 packages and Linux (32 and 64 bits) packages.

xq114 · 2024-09-18T15:11:05Z

packages/m/mkl/xmake.lua

@@ -109,7 +113,8 @@ package("mkl")
        package:add("links", package:is_arch("x64", "x86_64") and "mkl_blas95_" .. suffix or "mkl_blas95")
        package:add("links", package:is_arch("x64", "x86_64") and "mkl_lapack95_" .. suffix or "mkl_lapack95")

-        if package:has_tool("cc", "gcc", "gxx") then
+        if (package:has_tool("cc", "gcc", "gxx") or 
+            package:has_tool("fc", "gfortran")) then


Here we should check package:has_tool("cc", "cl", "clang-cl") first, otherwise on windows if there is gfortran installed it will lead to incorrect link flags

Basically it should also detect Apple clang since Apple clang do not support link groups. But I don't know how to tell Apple clang from llvm clang using package:has_tool() api. @waruqi any idea?

use package:has_flags() instead of has_tool?

Is there any other package I could use as reference for the correct test structure? The original versions of fetch.lua and xmake.lua had only this test.
Couldn't we use instead a test on toolkind and the type of linker, as done for instance in the OpenMP package (ref:

xmake-repo/packages/o/openmp/xmake.lua

Line 60 in 9286276

for _, toolkind in ipairs({"ld", "fcld"}) do

)? A problem I can see for our usage is that there seems to be currently some issue with the support of fcldflags (see #5255)

you can try, test linker will be better.

OK. Should I use for now ldflags for the fortran part?

fcldflags (binary) fcshflags (shared)

xmake-repo/packages/o/openmp/xmake.lua

Lines 76 to 78 in f008565

elseif package:has_tool(toolkind, "gfortran") then

result.fcldflags = "-fopenmp"

result.fcshflags = "-fopenmp"

It seems the new test has improved the results of the tests (in the meantime I discovered I did not need to load add_configs for fetch.lua for the 'runtime' test to work), but it still fails on Mac OS X x86-64.
Is it possible that the LLVM linker on Apple is also recognized as GCC?
Should I replace the test:
if (package:has_tool(toolkind, "gcc", "gxx") or package:has_tool(toolkind, "gfortran")) then
with something of the form:

if (not package:has_tool(toolkind, "clang", "clangxx") and (package:has_tool(toolkind, "gcc", "gxx") or package:has_tool(toolkind, "gfortran"))) then

?

For Fortran, I had to replace fcldflags with ldflags since the support of the former seems broken up to xmake xmake v2.9.5+dev.b938556. Using ldflags is necessary to compile Fortran programs with GFortran on Linux AMD64. I have commented out the fcldflags and fcshflags for testing purposes in the package source.

Altina-oz · 2024-09-19T03:31:20Z

The intel repository in pip is still alive

https://pypi.org/project/mkl/#files

https://pypi.org/project/mkl-static/#files

https://pypi.org/project/mkl-devel/#files

https://pypi.org/project/mkl-include/#files

(However the format is ".whl" thus needs to unzip a sub-directory(filename.whl/$version.data/) inside it)

The tests to set the linker flags are now based on "ld/fcld" instead of the compilers "cc/fc" to avoid problems with toolchains combining different tools.

jbloino · 2024-09-19T12:21:12Z

The intel repository in pip is still alive

https://pypi.org/project/mkl/#files

https://pypi.org/project/mkl-static/#files

https://pypi.org/project/mkl-devel/#files

https://pypi.org/project/mkl-include/#files

(However the format is ".whl" thus needs to unzip a sub-directory(filename.whl/$version.data/) inside it)

Thank you very much for the information. I am not familiar enough with the internal tools of xmake to do this correctly.
The motivation for this PR was to add support for Fortran/GFortran, that we use to develop codes for numerical simulations.
The patch to the repository was an extra necessary to be able to link and test the package. Searching the forums, we found this alternative that required minimal changes to the existing xmake-repo package and seemed the method recommended by Intel for conda users (https://community.intel.com/t5/Intel-Integrated-Performance/Problems-installing-with-conda-HTTP-403-FORBIDDEN/m-p/1631577/highlight/true#M28831). I do not know what would be the best/most robust strategy for the fetching.

waruqi · 2024-09-19T14:29:24Z

packages/m/mkl/fetch.lua

@@ -1,6 +1,8 @@
 import("lib.detect.find_path")
 import("lib.detect.find_library")

+-- add_configs("runtime",      {description = "Set MKL runtime for gcc/clang like compilers.", default = "default", type = "string", values = {"default", "custom"}})


remove unused comments

waruqi · 2024-09-19T14:29:40Z

packages/m/mkl/fetch.lua

-    else
-        table.join2(result.links, group)
+    for _, toolkind in ipairs({"ld", "fcld"}) do
+        -- if package:config("runtime") == "default" then


remove comment

waruqi · 2024-09-19T14:30:03Z

packages/m/mkl/fetch.lua

-        table.join2(result.links, group)
+    for _, toolkind in ipairs({"ld", "fcld"}) do
+        -- if package:config("runtime") == "default" then
+            if (package:has_tool(toolkind, "gcc", "gxx") or package:has_tool(toolkind, "gfortran")) then


remove () in if () then

waruqi · 2024-09-19T14:30:11Z

packages/m/mkl/fetch.lua

+                    result.shflags = table.concat(flags, " ")
+                else
+                    -- result.fcldflags = table.concat(flags, " ")
+                    -- result.fcshflags = table.concat(flags, " ")


remove comment

waruqi · 2024-09-19T14:30:59Z

packages/m/mkl/xmake.lua

@@ -3,90 +3,96 @@ package("mkl")
    set_homepage("https://software.intel.com/content/www/us/en/develop/tools/oneapi/components/onemkl.html")
    set_description("Intel® oneAPI Math Kernel Library")

+    add_configs("runtime",      {description = "Set MKL runtime for gcc/clang like compilers.", default = "default", type = "string", values = {"default", "custom"}})


runtime is builtin configuration, please rename to mkl_runtime

Sorry, I had not understood this command in the OpenMP package. At this point, there is no real need for this setup in MKL in my opinion, especially considering how complicated the configuration of the linker is. This should be set through the other parameters, so I have removed the reference.

waruqi · 2024-09-19T14:31:47Z

packages/m/mkl/xmake.lua

-                package:add("links", "mkl_gnu_thread", "gomp")
+        for _, toolkind in ipairs({"ld", "fcld"}) do
+            if package:config("runtime") == "default" then
+                if (package:has_tool(toolkind, "gcc", "gxx") or package:has_tool(toolkind, "gfortran")) then


remove ()

waruqi · 2024-09-19T14:31:55Z

packages/m/mkl/xmake.lua

+                        var_ldflags = 'ldflags'
+                        var_shflags = 'shflags'
+                        -- var_ldflags = 'fcldflags'
+                        -- var_shflags = 'fcshflags'


remove comment

waruqi · 2024-09-19T14:32:10Z

packages/m/mkl/xmake.lua

+            if package:config("runtime") == "default" then
+                if (package:has_tool(toolkind, "gcc", "gxx") or package:has_tool(toolkind, "gfortran")) then
+                    if package:has_tool(toolkind, "gcc", "gxx") then
+                        var_ldflags = 'ldflags'


use "" instead of ''

and please define it as local variable.

local var_ldflags

I should have applied all the corrections. I had to change slightly the layout of the code to have the variables available in the parent block.

* Use of "" instead of '' for strings. * Removal of unnecessary parentheses. * Use of "local" for local variables. * Removed unnecessary comments.

Previous patches incorrectly referred to the runtime configuration, which is supposed to be built in and not set in package.

waruqi · 2024-09-20T13:11:11Z

packages/m/mkl/fetch.lua

-        for _, lib in ipairs(group) do
-            result.ldflags = result.ldflags .. "-l" .. lib .. " "
+    for _, toolkind in ipairs({"ld", "fcld"}) do
+        if package:has_tool(toolkind, "gcc", "gxx") or package:has_tool(toolkind, "gfortran") then


package:has_tool(toolkind, "gcc", "gxx", "gfortran")

waruqi · 2024-09-20T13:14:23Z

packages/m/mkl/fetch.lua

+            else
+                result.ldflags = table.concat(flags, " ")
+                result.shflags = table.concat(flags, " ")
+            end


please simplify repeat table.concat, we need not call table.concat

if package:has_tool(toolkind, "gfortran") then result.fcldflags = flags result.fcshflags = flags else result.ldflags = flags result.shflags = flags end

waruqi · 2024-09-20T13:15:11Z

packages/m/mkl/fetch.lua

+                result.ldflags = table.concat(flags, " ")
+                result.shflags = table.concat(flags, " ")
+            else
+                result.ldflags = table.concat(flags, " ")


waruqi · 2024-09-20T13:17:34Z

packages/m/mkl/xmake.lua

-            elseif threading == "gomp" then
-                package:add("links", "mkl_gnu_thread", "gomp")
+        for _, toolkind in ipairs({"ld", "fcld"}) do
+            if package:has_tool(toolkind, "gcc", "gxx") or package:has_tool(toolkind, "gfortran") then


waruqi · 2024-09-20T13:17:55Z

packages/m/mkl/xmake.lua

+                local var_shflags = "shflags"
+                if package:has_tool(toolkind, "gfortran") then
+                    var_ldflags = "ldflags"
+                    var_shflags = "shflags"


fcldflags, fcshflags?

The problem is that the package fails with toolchain=gfortran because the linker flags are not properly set. This gives an error for both OpenMP (#5255) and MKL packages. If a special setting is necessary to get this to work with GFortran, could you send it to me, please?
We use xmake --toolchain=gfortran. A minimum example is provided in the bug report #5255 .
The most recent tests were done with xmake v2.9.5+dev.9563841

I'll take a look at this issue, but need to wait a few days as I've been busy in these days.

No problem, I am currently including the other changes you asked and will make the final changes as necessary, thank you very much!

Regarding the failure with Mac Intel x86_64, is there a workaround to make it work? Should we explicitly exclude clang with a condition like:

if not package:has_tool(toolkind, "clang", "clangxx") and package:has_tool(toolkind, "gcc", "gxx", "gfortran") then

I have supported fcldflags, you can try #5255 (comment)

It fails during the configuration:

The verbose file does not provide much further insight.

I have pushed the latest changes to the package with the edits you asked. This version works fine for gcc/clang toolchains but fails with gfortran (Linux Debian testing, latest version of xmake from the dev branch).

but it works on ci.

Is there any fortran/gfortran test in the test suite? From what I could see, only clang and gcc are tested, so the logic for gfortran is never controlled.

The fortran code test is not currently supported. Can you provide a test example project, I will try it.

waruqi · 2024-09-20T13:18:24Z

packages/m/mkl/xmake.lua

+                end
+                table.insert(flags, "-lmkl_core")
+                table.insert(flags, "-Wl,--end-group")
+                package:add(var_ldflags, table.concat(flags, " "))


we need not call table.concat

The variables for the linker with GFortran are now set to fcldflags and fcshflags following the latest changes in xmake. Some redundant or awkward constructs were also fixed.

waruqi · 2024-09-23T02:05:52Z

please wait for this patch xmake-io/xmake#5649

then you can add

    on_test(function (package)
        import("lib.detect.find_tool")
        if package.check_fcsnippets and find_tool("gfortran") then
            assert(package:check_fcsnippets({test = [[
    program hello
      print *, "Hello World!"
    end program hello
            ]]}))
        end
    end)

to test fortan code and links.

Added Fortran test if gfortran available.

waruqi · 2024-09-23T07:55:58Z

/usr/bin/gfortran -c -m64 -isystem /home/runner/.xmake/packages/m/mkl/2022.2.0+8748/49ba34e91b0e47908a2377d1814fc943/include -isystem /home/runner/.xmake/packages/t/tbb/2021.12.0/df9ece8dadfc4b68be28a7546372f41e/include -o /tmp/.xmake1001/240923/_0197CE630E2F4730807622F18DBBE540.o /tmp/.xmake1001/240923/_FBF1AB2116944858B1A7CAC521022F86.f90
/usr/bin/gfortran -o /tmp/.xmake1001/240923/_0197CE630E2F4730807622F18DBBE540.b /tmp/.xmake1001/240923/_0197CE630E2F4730807622F18DBBE540.o -m64 -Wl,--start-group -lmkl_intel_lp64 -lmkl_tbb_thread -lmkl_core -Wl,--end-group -L/home/runner/.xmake/packages/m/mkl/2022.2.0+8748/49ba34e91b0e47908a2377d1814fc943/lib -L/home/runner/.xmake/packages/t/tbb/2021.12.0/df9ece8dadfc4b68be28a7546372f41e/lib -lmkl_blas95_lp64 -lmkl_lapack95_lp64 -ltbb -ltbbmalloc -ltbbmalloc_proxy -lpthread -ldl
checking for fortran links(mkl_blas95_lp64, mkl_lapack95_lp64, tbb, tbbmalloc, tbbmalloc_proxy, pthread, dl)
checking for fortran snippet(test)
checkinfo: ...gramdir/core/sandbox/modules/import/core/tool/linker.lua:75: @programdir/core/sandbox/modules/os.lua:273: /usr/bin/ld: /home/runner/.xmake/packages/m/mkl/2022.2.0+8748/49ba34e91b0e47908a2377d1814fc943/lib/libmkl_tbb_thread.a(tbb_driver.o): undefined reference to symbol '__cxa_pure_virtual@@CXXABI_1.3'
/usr/bin/ld: /lib/x86_64-linux-gnu/libstdc++.so.6: error adding symbols: DSO missing from command line
collect2: error: ld returned 1 exit status

any idea?

jbloino · 2024-09-23T08:19:09Z

/usr/bin/gfortran -c -m64 -isystem /home/runner/.xmake/packages/m/mkl/2022.2.0+8748/49ba34e91b0e47908a2377d1814fc943/include -isystem /home/runner/.xmake/packages/t/tbb/2021.12.0/df9ece8dadfc4b68be28a7546372f41e/include -o /tmp/.xmake1001/240923/_0197CE630E2F4730807622F18DBBE540.o /tmp/.xmake1001/240923/_FBF1AB2116944858B1A7CAC521022F86.f90
/usr/bin/gfortran -o /tmp/.xmake1001/240923/_0197CE630E2F4730807622F18DBBE540.b /tmp/.xmake1001/240923/_0197CE630E2F4730807622F18DBBE540.o -m64 -Wl,--start-group -lmkl_intel_lp64 -lmkl_tbb_thread -lmkl_core -Wl,--end-group -L/home/runner/.xmake/packages/m/mkl/2022.2.0+8748/49ba34e91b0e47908a2377d1814fc943/lib -L/home/runner/.xmake/packages/t/tbb/2021.12.0/df9ece8dadfc4b68be28a7546372f41e/lib -lmkl_blas95_lp64 -lmkl_lapack95_lp64 -ltbb -ltbbmalloc -ltbbmalloc_proxy -lpthread -ldl
checking for fortran links(mkl_blas95_lp64, mkl_lapack95_lp64, tbb, tbbmalloc, tbbmalloc_proxy, pthread, dl)
checking for fortran snippet(test)
checkinfo: ...gramdir/core/sandbox/modules/import/core/tool/linker.lua:75: @programdir/core/sandbox/modules/os.lua:273: /usr/bin/ld: /home/runner/.xmake/packages/m/mkl/2022.2.0+8748/49ba34e91b0e47908a2377d1814fc943/lib/libmkl_tbb_thread.a(tbb_driver.o): undefined reference to symbol '__cxa_pure_virtual@@CXXABI_1.3'
/usr/bin/ld: /lib/x86_64-linux-gnu/libstdc++.so.6: error adding symbols: DSO missing from command line
collect2: error: ld returned 1 exit status

any idea?

I am investigating. I have already found one problem:

with --platform=clang, the test fails because it complains it cannot find fc:

> /usr/bin/clang -c -Qunused-arguments -m64 -isystem /home/julien/.xmake/packages/m/mkl/2024.2.2+15/ed3b9f2047bc46e7a60eaa2939fb1cb4/include -isystem /home/julien/.xmake/packages/t/tbb/2021.12.0/640883599fae429b998f3570747c09ff/include -o /tmp/.xmake1000/240923/_BD89E622906F46008C3B8E70365C2800.o /tmp/.xmake1000/240923/_E262C6D404D948D887E7C7E6A42797C7.c
checking for flags (-fdiagnostics-color=always) ... ok
> clang "-fdiagnostics-color=always" "-Qunused-arguments" "-m64"
  => installing mkl .. (1/clang) > /usr/bin/clang++ -o /tmp/.xmake1000/240923/_BD89E622906F46008C3B8E70365C2800.b /tmp/.xmake1000/240923/_BD89E622906F46008C3B8E70365C2800.o -m64 -L/home/julien/.xmake/packages/m/mkl/2024.2.2+15/ed3b9f2047bc46e7a60eaa2939fb1cb4/lib -L/home/julien/.xmake/packages/t/tbb/2021.12.0/640883599fae429b998f3570747c09ff/lib -lmkl_blas95_lp64 -lmkl_lapack95_lp64 -lmkl_intel_lp64 -lmkl_tbb_thread -lmkl_core -ltbb -ltbbmalloc -ltbbmalloc_proxy -lpthread -ldl
> checking for c includes(mkl_cblas.h)
> checking for c links(mkl_blas95_lp64, mkl_lapack95_lp64, mkl_intel_lp64, mkl_tbb_thread, mkl_core, tbb, tbbmalloc, tbbmalloc_proxy, pthread, dl)
> checking for c snippet(test)
> checking for fortran links(mkl_blas95_lp64, mkl_lapack95_lp64, mkl_intel_lp64, mkl_tbb_thread, mkl_core, tbb, tbbmalloc, tbbmalloc_proxy, pthread, dl)
> checking for fortran snippet(test)
checkinfo: ...amdir/core/sandbox/modules/import/core/tool/compiler.lua:37: cannot get program for fc
stack traceback:
    [C]: in function 'error'
    [@programdir/core/base/os.lua:1004]:
    [...amdir/core/sandbox/modules/import/core/tool/compiler.lua:37]: in function 'load'
    [...amdir/core/sandbox/modules/import/core/tool/compiler.lua:60]: in function 'compargv'
    [...amdir/core/sandbox/modules/import/core/tool/compiler.lua:44]: in function 'compcmd'
    [@programdir/modules/lib/detect/check_fcsnippets.lua:104]:

error: .../julien/dev/contribs/xmake-repo/packages/m/mkl/xmake.lua:185: ...amdir/core/sandbox/modules/import/core/tool/compiler.lua:37: cannot get program for fc
stack traceback:
    [C]: in function 'error'
    [@programdir/core/base/os.lua:1004]:
    [...amdir/core/sandbox/modules/import/core/tool/compiler.lua:37]: in function 'load'
    [...amdir/core/sandbox/modules/import/core/tool/compiler.lua:60]: in function 'compargv'
    [...amdir/core/sandbox/modules/import/core/tool/compiler.lua:44]: in function 'compcmd'
    [@programdir/modules/lib/detect/check_fcsnippets.lua:104]:

  => install mkl 2024.2.2+15 .. failed
error: @programdir/core/main.lua:329: @programdir/modules/async/runjobs.lua:325: .../modules/private/action/require/impl/actions/install.lua:494: install failed!
stack traceback:
    [C]: in function 'error'
    [@programdir/core/base/os.lua:1004]:
    [.../modules/private/action/require/impl/actions/install.lua:494]: in function 'catch'
    [@programdir/core/sandbox/modules/try.lua:123]: in function 'try'
    [.../modules/private/action/require/impl/actions/install.lua:361]:
    [...modules/private/action/require/impl/install_packages.lua:496]: in function 'jobfunc'
    [@programdir/modules/async/runjobs.lua:241]:

stack traceback:
        [C]: in function 'error'
        @programdir/core/base/os.lua:1004: in function 'os.raiselevel'
        (...tail calls...)
        @programdir/core/main.lua:329: in upvalue 'cotask'
        @programdir/core/base/scheduler.lua:406: in function <@programdir/core/base/scheduler.lua:399>
error: execv(xmake require -f -y --build -D --extra={configs={shared=false}} mkl) failed(255)

It seems that the test on the fcsnippet gives a false positive.

I myself found that the test could fail because it looks for gfortran but will use whatever is the default for the Fortran compiler, in my case nvfortran.

The test on the presence of gfortran may not guarantee the linker is properly set. The new test check that fcld is also set, for now to gfortran since the test on the Fortran compiler only considers this compiler. Note: this way, fcsnippets are only run if --toolchain=gfortran is set.

jbloino · 2024-09-25T16:29:34Z

For now, I was able to identify a few issues with the current setup:

on some installations where GFortran is present but not the Fortran compiler set by default, the test could fail. Adding a test on fcld to be gfortran seemed to fix the problem, but the Fortran test is ignored. I had this kind of issue when testing with --toolchain=clang.
for some reason, the default linker (ld) for the Fortran tests seems to be set to g++. In this case, libstdc++ is used but libstdc++.so.6 seems to be missing some symbols necessary until mkl 2024.2: with gcc, all versions of MKL since 2022 are supported, but with g++, only the latest versions seem to not give any problem. I did not find a way to change the linker for the fcsnippets. How could this be done? The test seems to work if older versions are removed, but this could break some builds that require an older versions of MKL.

waruqi · 2024-09-26T14:30:47Z

for some reason, the default linker (ld) for the Fortran tests seems to be set to g++. In this case, libstdc++ is used but libstdc++.so.6 seems to be missing some symbols necessary until mkl 2024.2: with gcc, all versions of MKL since 2022 are supported, but with g++, only the latest versions seem to not give any problem. I did not find a way to change the linker for the fcsnippets. How could this be done? The test seems to work if older versions are removed, but this could break some builds that require an older versions of MKL.

This may be caused by the use of precompiled mkl binary libraries. Binary libraries under Linux may have compatibility issues.

and it works on my linux machine.

waruqi · 2024-09-26T14:33:27Z

on macOS

/usr/local/bin/gfortran -o /var/folders/jq/0ryt2n7x2h9_f002rzt_hjj00000gn/T/.xmake501/240925/_8E5236E76F5742308A6E81FCFEB67960.b /var/folders/jq/0ryt2n7x2h9_f002rzt_hjj00000gn/T/.xmake501/240925/_8E5236E76F5742308A6E81FCFEB67960.o -m64 -L/Users/runner/.xmake/packages/m/mkl/2023.2.0+49499/ee39b681f4474c1988b1fbcdc0887b0f/lib -L/Users/runner/.xmake/packages/t/tbb/2021.12.0/ab0a93b2d4e54005995f4306fe36daae/lib -lmkl_blas95_lp64 -lmkl_lapack95_lp64 -lmkl_intel_lp64 -lmkl_tbb_thread -lmkl_core -ltbb -ltbbmalloc -ltbbmalloc_proxy
checking for fortran links(mkl_blas95_lp64, mkl_lapack95_lp64, mkl_intel_lp64, mkl_tbb_thread, mkl_core, tbb, tbbmalloc, tbbmalloc_proxy)
checking for fortran snippet(test)
checkinfo: ...gramdir/core/sandbox/modules/import/core/tool/linker.lua:75: @programdir/core/sandbox/modules/os.lua:273: ld: warning: dylib (/Users/runner/.xmake/packages/t/tbb/2021.12.0/ab0a93b2d4e54005995f4306fe36daae/lib/libtbb.dylib) was built for newer macOS version (12.7) than being linked (12.0)
ld: warning: dylib (/Users/runner/.xmake/packages/t/tbb/2021.12.0/ab0a93b2d4e54005995f4306fe36daae/lib/libtbbmalloc.dylib) was built for newer macOS version (12.7) than being linked (12.0)
ld: warning: dylib (/Users/runner/.xmake/packages/t/tbb/2021.12.0/ab0a93b2d4e54005995f4306fe36daae/lib/libtbbmalloc_proxy.dylib) was built for newer macOS version (12.7) than being linked (12.0)
Undefined symbols for architecture x86_64:
"operator delete(void*)", referenced from:
tbb::detail::d1::start_for<tbb::detail::d1::blocked_range, (anonymous namespace)::ThreadDriver, tbb::detail::d1::simple_partitioner const>::~start_for() in libmkl_tbb_thread.a(tbb_driver.o)
tbb::detail::d1::start_for<tbb::detail::d1::blocked_range, (anonymous namespace)::ThreadDriver, tbb::detail::d1::affinity_partitioner>::~start_for() in libmkl_tbb_thread.a(tbb_driver.o)
_mkl_blas_delete_affinity_partitioner in libmkl_tbb_thread.a(tbb_driver.o)
"operator delete(void*, std::align_val_t)", referenced from:
tbb::detail::d1::task::~task() in libmkl_tbb_thread.a(tbb_driver.o)
"operator new(unsigned long)", referenced from:
_mkl_blas_new_affinity_partitioner in libmkl_tbb_thread.a(tbb_driver.o)
"___cxa_pure_virtual", referenced from:
vtable for tbb::detail::d1::task in libmkl_tbb_thread.a(tbb_driver.o)
ld: symbol(s) not found for architecture x86_64

maybe we need add c++ links for frotran test, because it will link some c++ libs.

waruqi · 2024-09-26T14:42:11Z

It seems that all are compatibility issues caused by mkl binary libraries.

waruqi · 2024-09-26T15:01:29Z

if I use g++ as linker, some fortran libs will be missing.

> /usr/bin/g++ -o /tmp/.xmake999/240926/_CA59D77C6B80423084BB868ED6D8C940.b /tmp/.xmake999/240926/_CA59D77C6B80423084BB868ED6D8C940.o -m64 -Wl,--start-group -lmkl_intel_lp64 -lmkl_tbb_thread -lmkl_core -Wl,--end-group -L/home/ruki/.xmake/packages/m/mkl/2024.2.2+15/52a38912be4e4ed1bdfef80bb11ed833/lib -L/home/ruki/.xmake/packages/t/tbb/2021.12.0/df9ece8dadfc4b68be28a7546372f41e/lib -lmkl_blas95_lp64 -lmkl_lapack95_lp64 -ltbb -ltbbmalloc -ltbbmalloc_proxy -lpthread -ldl
/usr/bin/ld: /tmp/.xmake999/240926/_CA59D77C6B80423084BB868ED6D8C940.o: in function `main':
_FBF1AB2116944858B1A7CAC521022F86.f90:(.text+0xa2): undefined reference to `_gfortran_set_args'
/usr/bin/ld: _FBF1AB2116944858B1A7CAC521022F86.f90:(.text+0xb6): undefined reference to `_gfortran_set_options'
collect2: error: ld returned 1 exit status

jbloino · 2024-09-26T15:06:49Z

on macOS

/usr/local/bin/gfortran -o /var/folders/jq/0ryt2n7x2h9_f002rzt_hjj00000gn/T/.xmake501/240925/_8E5236E76F5742308A6E81FCFEB67960.b /var/folders/jq/0ryt2n7x2h9_f002rzt_hjj00000gn/T/.xmake501/240925/_8E5236E76F5742308A6E81FCFEB67960.o -m64 -L/Users/runner/.xmake/packages/m/mkl/2023.2.0+49499/ee39b681f4474c1988b1fbcdc0887b0f/lib -L/Users/runner/.xmake/packages/t/tbb/2021.12.0/ab0a93b2d4e54005995f4306fe36daae/lib -lmkl_blas95_lp64 -lmkl_lapack95_lp64 -lmkl_intel_lp64 -lmkl_tbb_thread -lmkl_core -ltbb -ltbbmalloc -ltbbmalloc_proxy
checking for fortran links(mkl_blas95_lp64, mkl_lapack95_lp64, mkl_intel_lp64, mkl_tbb_thread, mkl_core, tbb, tbbmalloc, tbbmalloc_proxy)
checking for fortran snippet(test)
checkinfo: ...gramdir/core/sandbox/modules/import/core/tool/linker.lua:75: @programdir/core/sandbox/modules/os.lua:273: ld: warning: dylib (/Users/runner/.xmake/packages/t/tbb/2021.12.0/ab0a93b2d4e54005995f4306fe36daae/lib/libtbb.dylib) was built for newer macOS version (12.7) than being linked (12.0)
ld: warning: dylib (/Users/runner/.xmake/packages/t/tbb/2021.12.0/ab0a93b2d4e54005995f4306fe36daae/lib/libtbbmalloc.dylib) was built for newer macOS version (12.7) than being linked (12.0)
ld: warning: dylib (/Users/runner/.xmake/packages/t/tbb/2021.12.0/ab0a93b2d4e54005995f4306fe36daae/lib/libtbbmalloc_proxy.dylib) was built for newer macOS version (12.7) than being linked (12.0)
Undefined symbols for architecture x86_64:
"operator delete(void*)", referenced from:
tbb::detail::d1::start_for<tbb::detail::d1::blocked_range, (anonymous namespace)::ThreadDriver, tbb::detail::d1::simple_partitioner const>::~start_for() in libmkl_tbb_thread.a(tbb_driver.o)
tbb::detail::d1::start_for<tbb::detail::d1::blocked_range, (anonymous namespace)::ThreadDriver, tbb::detail::d1::affinity_partitioner>::~start_for() in libmkl_tbb_thread.a(tbb_driver.o)
_mkl_blas_delete_affinity_partitioner in libmkl_tbb_thread.a(tbb_driver.o)
"operator delete(void*, std::align_val_t)", referenced from:
tbb::detail::d1::task::~task() in libmkl_tbb_thread.a(tbb_driver.o)
"operator new(unsigned long)", referenced from:
_mkl_blas_new_affinity_partitioner in libmkl_tbb_thread.a(tbb_driver.o)
"___cxa_pure_virtual", referenced from:
vtable for tbb::detail::d1::task in libmkl_tbb_thread.a(tbb_driver.o)
ld: symbol(s) not found for architecture x86_64

maybe we need add c++ links for frotran test, because it will link some c++ libs.

I tried to do this but failed. How should I add them in the snippets?

jbloino · 2024-09-26T15:07:49Z

if I use g++ as linker, some fortran libs will be missing.

> /usr/bin/g++ -o /tmp/.xmake999/240926/_CA59D77C6B80423084BB868ED6D8C940.b /tmp/.xmake999/240926/_CA59D77C6B80423084BB868ED6D8C940.o -m64 -Wl,--start-group -lmkl_intel_lp64 -lmkl_tbb_thread -lmkl_core -Wl,--end-group -L/home/ruki/.xmake/packages/m/mkl/2024.2.2+15/52a38912be4e4ed1bdfef80bb11ed833/lib -L/home/ruki/.xmake/packages/t/tbb/2021.12.0/df9ece8dadfc4b68be28a7546372f41e/lib -lmkl_blas95_lp64 -lmkl_lapack95_lp64 -ltbb -ltbbmalloc -ltbbmalloc_proxy -lpthread -ldl
/usr/bin/ld: /tmp/.xmake999/240926/_CA59D77C6B80423084BB868ED6D8C940.o: in function `main':
_FBF1AB2116944858B1A7CAC521022F86.f90:(.text+0xa2): undefined reference to `_gfortran_set_args'
/usr/bin/ld: _FBF1AB2116944858B1A7CAC521022F86.f90:(.text+0xb6): undefined reference to `_gfortran_set_options'
collect2: error: ld returned 1 exit status

It should be possible to fix these errors with -lgfortran, which adds the Fortran symbols for gcc.

jbloino · 2024-09-27T07:27:20Z

It seems that the Fortran tests (fcsnippets) use the linker defined as ld. Typically, we use directly the Fortran compiler as the linker as well. Could xmake use fcld for the linker and how could we set it up?

xq114 · 2024-10-28T17:14:52Z

I'm going to split this issue into 2 parts since the fortran things are still not fully solved, and fixing the broken urls is more important. See #5602

Julien Bloino and others added 2 commits September 17, 2024 09:15

Merge branch 'xmake-io:dev' into mkl_new_url

c76aa5c

waruqi requested review from xq114 and SirLynix September 17, 2024 13:24

Fixed deprecated packages in new repository.

532d08d

This patch restores support of Intel Mac versions

5c6d269

The repository was also available in the new repository. Versions listed in the package and not available in the repository have been removed or replaced with more recent ones from the same year.

Fixed URLs to some packages.

adb9d01

Fixed URLs to some Windows 32 packages and Linux (32 and 64 bits) packages.

xq114 requested changes Sep 18, 2024

View reviewed changes

Check on linker instead of compiler to set flags

154419a

The tests to set the linker flags are now based on "ld/fcld" instead of the compilers "cc/fc" to avoid problems with toolchains combining different tools.

waruqi reviewed Sep 19, 2024

View reviewed changes

Julien Bloino added 2 commits September 20, 2024 09:47

Fixed code layout to match conventions

1b75df6

* Use of "" instead of '' for strings. * Removal of unnecessary parentheses. * Use of "local" for local variables. * Removed unnecessary comments.

Remove misleading reference to runtime config.

e1846a6

Previous patches incorrectly referred to the runtime configuration, which is supposed to be built in and not set in package.

waruqi reviewed Sep 20, 2024

View reviewed changes

Switch to fcldflags/fcshflags for GFortran

e55dc93

The variables for the linker with GFortran are now set to fcldflags and fcshflags following the latest changes in xmake. Some redundant or awkward constructs were also fixed.

waruqi mentioned this pull request Sep 23, 2024

add package:check_fcsnippets xmake-io/xmake#5649

Merged

Added Fortran test for the MKL package

039c45b

Added Fortran test if gfortran available.

Update xmake.lua

02faa26

xq114 mentioned this pull request Oct 28, 2024

fix mkl url #5602

Merged

	elseif package:has_tool(toolkind, "gfortran") then
	result.fcldflags = "-fopenmp"
	result.fcshflags = "-fopenmp"

Add support of GFortran in MKL package #5258

Are you sure you want to change the base?

Add support of GFortran in MKL package #5258

Conversation

jbloino commented Sep 17, 2024

xq114 commented Sep 18, 2024

jbloino commented Sep 18, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Altina-oz commented Sep 19, 2024 • edited Loading

jbloino commented Sep 19, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jbloino Sep 20, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

waruqi commented Sep 23, 2024

waruqi commented Sep 23, 2024

jbloino commented Sep 23, 2024

jbloino commented Sep 25, 2024

waruqi commented Sep 26, 2024

waruqi commented Sep 26, 2024

waruqi commented Sep 26, 2024

waruqi commented Sep 26, 2024

jbloino commented Sep 26, 2024

jbloino commented Sep 26, 2024

jbloino commented Sep 27, 2024

xq114 commented Oct 28, 2024

Altina-oz commented Sep 19, 2024 •

edited

Loading

jbloino Sep 20, 2024 •

edited

Loading