Build with NumPy 2

[gspowley]

Changes in this PR

• Remove numpy and pyarrow pins
• Remove python 3.8 (not supported by numpy 2) and add python 3.12
• Add changes for windows build in conda feedstock
• Remove unused Dockerfile and env file

See TileDB-Inc/tiledb-vcf-feedstock#138 for conda feedstock experiments using this branch.

[gspowley]

For [sc-53799].

[jdblischak]

Also need to remove pins from ci/gha-win-env.yml. It is installing numpy<2 and pyarrow==9

TileDB-VCF/ci/gha-win-env.yml

Lines 14 to 17 in fd51152

	- numpy<2
	- pandas<2.0
	- pyarrow=9.0
	- pyarrow-hotfix

And while you are editing that file, could you please also update the tiledb pin?

TileDB-VCF/ci/gha-win-env.yml

Line 11 in fd51152

- tiledb=2.15

[gspowley]

Also need to remove pins from ci/gha-win-env.yml. It is installing numpy<2 and pyarrow==9

@jdblischak here are some failed windows build experiments with similar changes to ci/gha-win-env.yml:

• https://github.com/TileDB-Inc/TileDB-VCF/actions/runs/10588385072/job/29340808671
• https://github.com/TileDB-Inc/TileDB-VCF/actions/runs/10588513617/job/29341137814
• https://github.com/TileDB-Inc/TileDB-VCF/actions/runs/10588680628/job/29341573089

The windows feedstock is also broken - https://github.com/TileDB-Inc/tiledb-vcf-feedstock/runs/29369534110

I'll try a few more experiments. Let me know if you see any clues in the error logs above.

[jdblischak]

So the errors here are LNK2001 errors related to libfmt:

reader.obj : error LNK2019: unresolved external symbol "public: __cdecl fmt::v11::detail::locale_ref::locale_ref<class std::locale>(class std::locale const &)" (??$?0Vlocale@std@@@locale_ref@detail@v11@fmt@@QEAA@AEBVlocale@std@@@Z) referenced in function "class std::basic_string<char,struct std::char_traits<char>,class std::allocator<char> > __cdecl fmt::v11::vformat<class std::locale,0>(class std::locale const &,class fmt::v11::basic_string_view<char>,class fmt::v11::basic_format_args<class fmt::v11::context>)" (??$vformat@Vlocale@std@@$0A@@v11@fmt@@YA?AV?$basic_string@DU?$char_traits@D@std@@V?$allocator@D@2@@std@@AEBVlocale@3@V?$basic_string_view@D@01@V?$basic_format_args@Vcontext@v11@fmt@@@01@@Z) [D:\a\TileDB-VCF\TileDB-VCF\libtiledbvcf-build\libtiledbvcf\src\tiledbvcf-bin.vcxproj] [D:\a\TileDB-VCF\TileDB-VCF\libtiledbvcf-build\libtiledbvcf.vcxproj]

These are different than the errors that are currently breaking tiledb-vcf-feedstock, which look like:

D:\bld\tiledb-vcf_1724188018821\_h_env\Library\include\fmt\base.h(458,28): error C2338: Unicode support requires compiling with /utf-8 [D:\bld\tiledb-vcf_1724188018821\work\libtiledbvcf-build\libtiledbvcf\src\TILEDB_VCF_OBJECTS.vcxproj] [D:\bld\tiledb-vcf_1724188018821\work\libtiledbvcf-build\libtiledbvcf.vcxproj]

Though these are also related to libfmt

[gspowley]

@jdblischak, @ihnorton as we saw in #778 (comment), the windows build is broken for reasons other that this PR.

At this point, I plan to merge PRs with failing windows builds.

[ihnorton]

Ok, SGTM. @dudoslav is looking at the windows issues.

[jdblischak]

So the errors here are LNK2001 errors related to libfmt:

A quick post-mortem on this. I had been very puzzled why the libfmt errors we have observed in tiledb-vcf-feedstock for months all of a sudden appeared in the Windows CI here in the TileDB-VCF repo. I think I have an explanation:

• By removing the pins pyarrow==9 and tiledb=2.15 in the conda environment file, a newer version of tiledb was able to be installed
• This new version of tiledb links to fmt/spdlog installed from conda, which explains why we started seeing similar errors as we do in the feedstock builds. That's also why the Windows build only started failing in this PR with commit 009ff67

Unfortunately I don't know how to fix this. But if we can figure out the libfmt linking problem, we can solve the Windows builds both in this repo and in the feedstock.

Here is how I arrived at this conclusion. I copy-pasted the output of micromamba list for the Windows CI from the last passing build in 2271db0 and the subsequent failing build in 57be5e4. Then I diffed them locally. The passing build didn't even have fmt/spdlog installed in the conda env, because tiledb 2.15 self-installed these during its superbuild.

$ diff vcf-win-pass.txt vcf-win-fail.txt | grep cmake
<     cmake                       3.29.6              h400e5d1_0             conda-forge
>     cmake                       3.30.5                 h400e5d1_0           conda-forge

$ diff vcf-win-pass.txt vcf-win-fail.txt | grep fmt
>     fmt                         11.0.2                 h7f575de_0           conda-forge

$ diff vcf-win-pass.txt vcf-win-fail.txt | grep spdlog
>     spdlog                      1.14.1                 h9f2357e_1           conda-forge

$ grep fmt vcf-win-*
vcf-win-fail.txt:    fmt                         11.0.2                 h7f575de_0           conda-forge

$ diff vcf-win-pass.txt vcf-win-fail.txt | grep tiledb
<     m2w64-htslib                1.19                h8a9f4b3_0             tiledb
<     tiledb                      2.15.4              h616f04a_1             conda-forge
<     tiledb-py                   0.21.6              py311h37c03f0_0        conda-forge
>     m2w64-htslib                1.20                   h8a9f4b3_0           tiledb
>     tiledb                      2.25.0                 h19c5691_18          conda-forge
>     tiledb-py                   0.31.1                 py312h195dda3_0      conda-forge

$ diff vcf-win-pass.txt vcf-win-fail.txt | grep pyarrow
<     pyarrow                     9.0.0               py311h6a6099b_49_cpu   conda-forge
<     pyarrow-hotfix              0.6                 pyhd8ed1ab_0           conda-forge
>     pyarrow                     17.0.0                 py312h7e22eef_1      conda-forge
>     pyarrow-core                17.0.0                 py312h6a9c419_1_cpu  conda-forge
>     pyarrow-hotfix              0.6                    pyhd8ed1ab_0         conda-forge

$ diff vcf-win-pass.txt vcf-win-fail.txt | grep vs
<     vs2015_runtime              14.40.33810         h3bf8584_20            conda-forge
<     vs2019_win-64               19.29.30139         he1865b1_20            conda-forge
<     vswhere                     3.1.4               h57928b3_0             conda-forge
>     vs2015_runtime              14.40.33810            h3bf8584_22          conda-forge
>     vs2019_win-64               19.29.30139            he1865b1_22          conda-forge
>     vswhere                     3.1.7                  h57928b3_0           conda-forge