Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Importing pyarrow on Windows fails when calling from Julia via PythonCall #1176

Open
1 task done
hhaensel opened this issue Sep 28, 2023 · 4 comments
Open
1 task done

Importing pyarrow on Windows fails when calling from Julia via PythonCall #1176

hhaensel opened this issue Sep 28, 2023 · 4 comments
Labels
bug

Comments

@hhaensel
Copy link

hhaensel commented Sep 28, 2023 •
edited
Loading

Solution to issue cannot be found in the documentation.

  • I checked the documentation.

Issue

When I try to import pyarrow via PythonCall from Julia, I receive an error if pyarrow is installed via conda.
MWE:
Create a clean environment:

conda create -n pyarrow_conda -c conda-forge pyarrow

and then execute the following Julia code

julia> ENV["JULIA_PYTHONCALL_EXE"] = "C:\\ProgramData\\mambaforge\\envs\\pyarrow_conda\\python.exe"
"C:\\ProgramData\\mambaforge\\envs\\pyarrow_conda\\python.exe"

julia> using PythonCall

julia> pyimport("pyarrow")
ERROR: Python: ImportError: DLL load failed while importing lib: Die angegebene Prozedur wurde nicht gefunden.
Python stacktrace:
 [1] <module>
   @ C:\ProgramData\mambaforge\envs\pyarrow_conda\Lib\site-packages\pyarrow\__init__.py:65
Stacktrace:
 [1] pythrow()
   @ PythonCall C:\Users\<user>\.julia\dev\PythonCall\src\err.jl:94
 [2] errcheck
   @ C:\Users\<user>\.julia\dev\PythonCall\src\err.jl:10 [inlined]
 [3] pyimport(m::String)
   @ PythonCall C:\Users\<user>\.julia\dev\PythonCall\src\concrete\import.jl:11
 [4] top-level scope
   @ REPL[3]:1

However, if I install pyarrow via pip, no error is produced.

julia> ENV["JULIA_PYTHONCALL_EXE"] = "C:\\ProgramData\\mambaforge\\envs\\pyarrow_pip\\python.exe"
"C:\\ProgramData\\mambaforge\\envs\\pyarrow_pip\\python.exe"

julia> using PythonCall

julia> pyimport("pyarrow")
Python: <module 'pyarrow' from 'C:\\ProgramData\\mambaforge\\envs\\pyarrow_pip\\Lib\\site-packages\\pyarrow\\__init__.py'>

On a ubuntu linux, no error occurs for conda installations of pyarrow.

The file __init.py is identical in both conda and pip installation.

Any hint where this could come from?

Installed packages

Difference between pyarrow_pip and pyarrow_conda:

pyarrow                   13.0.0          py311h6a6099b_4_cpu    conda-forge
pyarrow                   13.0.0                   pypi_0    pypi

#-----------------------------------------------------------------------

Detailed listings:


(pyarrow_conda) PS C:\Users\<user>> conda list
# packages in environment at C:\ProgramData\mambaforge\envs\pyarrow_conda:
#
# Name                    Version                   Build  Channel
aws-c-auth                0.7.3                hd125877_3    conda-forge
aws-c-cal                 0.6.2                h04c9df6_1    conda-forge
aws-c-common              0.9.0                hcfcfb64_0    conda-forge
aws-c-compression         0.2.17               h04c9df6_2    conda-forge
aws-c-event-stream        0.3.2                h495bb32_0    conda-forge
aws-c-http                0.7.12               h0890e15_1    conda-forge
aws-c-io                  0.13.32              h83b3346_3    conda-forge
aws-c-mqtt                0.9.6                h0fd1aac_0    conda-forge
aws-c-s3                  0.3.17               h9f49523_0    conda-forge
aws-c-sdkutils            0.1.12               h04c9df6_1    conda-forge
aws-checksums             0.1.17               h04c9df6_1    conda-forge
aws-crt-cpp               0.23.1               hfd73fcf_2    conda-forge
aws-sdk-cpp               1.11.156             h03febf0_2    conda-forge
bzip2                     1.0.8                h8ffe710_4    conda-forge
c-ares                    1.19.1               hcfcfb64_0    conda-forge
ca-certificates           2023.7.22            h56e8100_0    conda-forge
intel-openmp              2023.2.0         h57928b3_49503    conda-forge
krb5                      1.21.2               heb0366b_0    conda-forge
libabseil                 20230802.1      cxx17_h63175ca_0    conda-forge
libarrow                  13.0.0           h1e3473c_4_cpu    conda-forge
libblas                   3.9.0              18_win64_mkl    conda-forge
libbrotlicommon           1.1.0                hcfcfb64_0    conda-forge
libbrotlidec              1.1.0                hcfcfb64_0    conda-forge
libbrotlienc              1.1.0                hcfcfb64_0    conda-forge
libcblas                  3.9.0              18_win64_mkl    conda-forge
libcrc32c                 1.1.2                h0e60522_0    conda-forge
libcurl                   8.3.0                hd5e4a3a_0    conda-forge
libevent                  2.1.12               h3671451_1    conda-forge
libexpat                  2.5.0                h63175ca_1    conda-forge
libffi                    3.4.2                h8ffe710_5    conda-forge
libgoogle-cloud           2.12.0               h0a0a397_2    conda-forge
libgrpc                   1.57.0               h550f6bd_1    conda-forge
libhwloc                  2.9.3           default_haede6df_1009    conda-forge
libiconv                  1.17                 h8ffe710_0    conda-forge
liblapack                 3.9.0              18_win64_mkl    conda-forge
libprotobuf               4.23.4               hb8276f3_6    conda-forge
libsqlite                 3.43.0               hcfcfb64_0    conda-forge
libssh2                   1.11.0               h7dfc565_0    conda-forge
libthrift                 0.19.0               h06f6336_0    conda-forge
libutf8proc               2.8.0                h82a8f57_0    conda-forge
libxml2                   2.11.5               hc3477c8_1    conda-forge
libzlib                   1.2.13               hcfcfb64_5    conda-forge
lz4-c                     1.9.4                hcfcfb64_0    conda-forge
mkl                       2022.1.0           h6a75c08_874    conda-forge
numpy                     1.26.0          py311h0b4df5a_0    conda-forge
openssl                   3.1.3                hcfcfb64_0    conda-forge
orc                       1.9.0                h8dbeef6_2    conda-forge
pip                       23.2.1             pyhd8ed1ab_0    conda-forge
pthreads-win32            2.9.1                hfa6e2cd_3    conda-forge
pyarrow                   13.0.0          py311h6a6099b_4_cpu    conda-forge
python                    3.11.5          h2628c8c_0_cpython    conda-forge
python_abi                3.11                    4_cp311    conda-forge
re2                       2023.03.02           hd4eee63_0    conda-forge
setuptools                68.2.2             pyhd8ed1ab_0    conda-forge
snappy                    1.1.10               hfb803bf_0    conda-forge
tbb                       2021.10.0            h91493d7_1    conda-forge
tk                        8.6.13               hcfcfb64_0    conda-forge
tzdata                    2023c                h71feb2d_0    conda-forge
ucrt                      10.0.22621.0         h57928b3_0    conda-forge
vc                        14.3                h64f974e_17    conda-forge
vc14_runtime              14.36.32532         hdcecf7f_17    conda-forge
vs2015_runtime            14.36.32532         h05e6639_17    conda-forge
wheel                     0.41.2             pyhd8ed1ab_0    conda-forge
xz                        5.2.6                h8d14728_0    conda-forge
zstd                      1.5.5                h12be248_0    conda-forge

# -------------------------------------------------------------------------

(pyarrow_pip) PS C:\Users\<user>> conda list
# packages in environment at C:\ProgramData\mambaforge\envs\pyarrow_pip:
#
# Name                    Version                   Build  Channel
aws-c-auth                0.7.3                hd125877_3    conda-forge
aws-c-cal                 0.6.2                h04c9df6_1    conda-forge
aws-c-common              0.9.0                hcfcfb64_0    conda-forge
aws-c-compression         0.2.17               h04c9df6_2    conda-forge
aws-c-event-stream        0.3.2                h495bb32_0    conda-forge
aws-c-http                0.7.12               h0890e15_1    conda-forge
aws-c-io                  0.13.32              h83b3346_3    conda-forge
aws-c-mqtt                0.9.6                h0fd1aac_0    conda-forge
aws-c-s3                  0.3.17               h9f49523_0    conda-forge
aws-c-sdkutils            0.1.12               h04c9df6_1    conda-forge
aws-checksums             0.1.17               h04c9df6_1    conda-forge
aws-crt-cpp               0.23.1               hfd73fcf_2    conda-forge
aws-sdk-cpp               1.11.156             h03febf0_2    conda-forge
bzip2                     1.0.8                h8ffe710_4    conda-forge
c-ares                    1.19.1               hcfcfb64_0    conda-forge
ca-certificates           2023.7.22            h56e8100_0    conda-forge
intel-openmp              2023.2.0         h57928b3_49503    conda-forge
krb5                      1.21.2               heb0366b_0    conda-forge
libabseil                 20230802.1      cxx17_h63175ca_0    conda-forge
libarrow                  13.0.0           h1e3473c_4_cpu    conda-forge
libblas                   3.9.0              18_win64_mkl    conda-forge
libbrotlicommon           1.1.0                hcfcfb64_0    conda-forge
libbrotlidec              1.1.0                hcfcfb64_0    conda-forge
libbrotlienc              1.1.0                hcfcfb64_0    conda-forge
libcblas                  3.9.0              18_win64_mkl    conda-forge
libcrc32c                 1.1.2                h0e60522_0    conda-forge
libcurl                   8.3.0                hd5e4a3a_0    conda-forge
libevent                  2.1.12               h3671451_1    conda-forge
libexpat                  2.5.0                h63175ca_1    conda-forge
libffi                    3.4.2                h8ffe710_5    conda-forge
libgoogle-cloud           2.12.0               h0a0a397_2    conda-forge
libgrpc                   1.57.0               h550f6bd_1    conda-forge
libhwloc                  2.9.3           default_haede6df_1009    conda-forge
libiconv                  1.17                 h8ffe710_0    conda-forge
liblapack                 3.9.0              18_win64_mkl    conda-forge
libprotobuf               4.23.4               hb8276f3_6    conda-forge
libsqlite                 3.43.0               hcfcfb64_0    conda-forge
libssh2                   1.11.0               h7dfc565_0    conda-forge
libthrift                 0.19.0               h06f6336_0    conda-forge
libutf8proc               2.8.0                h82a8f57_0    conda-forge
libxml2                   2.11.5               hc3477c8_1    conda-forge
libzlib                   1.2.13               hcfcfb64_5    conda-forge
lz4-c                     1.9.4                hcfcfb64_0    conda-forge
mkl                       2022.1.0           h6a75c08_874    conda-forge
numpy                     1.26.0          py311h0b4df5a_0    conda-forge
openssl                   3.1.3                hcfcfb64_0    conda-forge
orc                       1.9.0                h8dbeef6_2    conda-forge
pip                       23.2.1             pyhd8ed1ab_0    conda-forge
pthreads-win32            2.9.1                hfa6e2cd_3    conda-forge
pyarrow                   13.0.0                   pypi_0    pypi
python                    3.11.5          h2628c8c_0_cpython    conda-forge
python_abi                3.11                    4_cp311    conda-forge
re2                       2023.03.02           hd4eee63_0    conda-forge
setuptools                68.2.2             pyhd8ed1ab_0    conda-forge
snappy                    1.1.10               hfb803bf_0    conda-forge
tbb                       2021.10.0            h91493d7_1    conda-forge
tk                        8.6.13               hcfcfb64_0    conda-forge
tzdata                    2023c                h71feb2d_0    conda-forge
ucrt                      10.0.22621.0         h57928b3_0    conda-forge
vc                        14.3                h64f974e_17    conda-forge
vc14_runtime              14.36.32532         hdcecf7f_17    conda-forge
vs2015_runtime            14.36.32532         h05e6639_17    conda-forge
wheel                     0.41.2             pyhd8ed1ab_0    conda-forge
xz                        5.2.6                h8d14728_0    conda-forge
zstd                      1.5.5                h12be248_0    conda-forge

Environment info

active environment : pyarrow_conda
    active env location : C:\ProgramData\mambaforge\envs\pyarrow_conda
            shell level : 4
       user config file : C:\Users\<user>\.condarc
 populated config files : C:\ProgramData\mambaforge\.condarc
                          C:\Users\<user>\.condarc
          conda version : 23.3.1
    conda-build version : not installed
         python version : 3.10.12.final.0
       virtual packages : __archspec=1=x86_64
                          __cuda=12.1=0
                          __win=0=0
       base environment : C:\ProgramData\mambaforge  (writable)
      conda av data dir : C:\ProgramData\mambaforge\etc\conda
  conda av metadata url : None
           channel URLs : https://conda.anaconda.org/conda-forge/win-64
                          https://conda.anaconda.org/conda-forge/noarch
          package cache : C:\ProgramData\mambaforge\pkgs
       envs directories : C:\ProgramData\mambaforge\envs
               platform : win-64
             user-agent : conda/23.3.1 requests/2.31.0 CPython/3.10.12 Windows/10 Windows/10.0.19045
          administrator : False
             netrc file : None
           offline mode : False
@hhaensel hhaensel added the bug label Sep 28, 2023
@h-vetinari
Copy link
Member

h-vetinari commented Sep 28, 2023 •
edited
Loading

Thanks for the report. Conda-forge builds arrow with more bells & whistles than the wheel builds for pip, but I don't see why things shouldn't be loadable. However, I don't see in your error logs which DLL actually fails to be loaded. Is there a way to get more verbose output of that failure?

Also, I have no experience with the vagaries of julia, perhaps someone from @conda-forge/julia already has an idea what's going on.

@hhaensel hhaensel mentioned this issue Sep 29, 2023
Open
@h-vetinari
Copy link
Member

Any update here @hhaensel?

@mkitti
Copy link

mkitti commented Dec 4, 2023

I'm going to note this is on Windows. conda-forge does not provide a Windows build of Julia.

@mkitti
Copy link

mkitti commented Dec 4, 2023

I'm a bit curious why one would be using pyarrow through PythonCall since there is also a Julia interface to Arrow:

https://arrow.apache.org/julia/dev/

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@mkitti @hhaensel @h-vetinari