-
Notifications
You must be signed in to change notification settings - Fork 207
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Circular imports for Python 3.8 on Windows if DLLs are already loaded #944
Comments
The issue is with Lines 43 to 46 in 52ad006
Starting with Python 3.8 dll directories must be added with add_dll_directory() on Windows. (#851) On Fionas appveyor, the import of import fiona.ogrext will fail as there are no DLL presents at the time of import, or fiona.ogrext is already loaded. But in the case of conda, the DLLs are already loaded. Thus the import of fiona.ogrext does not raise an ImportError and fiona.ogrext will import fiona.env, that again imports fiona._loading, which leads to a circular import. A quick fix could be to change the test import to something that does not import anything else from Fiona. E.g.
But this will be quite dangerous because with adding additional imports the problem can arise again. |
@rbuffat as we solve this, let's also move this code
into |
I created a PR in #945. If you have already implemented something feel free to close it. |
Fixed in Fiona 1.8.16 |
I'm leaving a comment here because it's the only Google result which matched my error exactly. Hopefully it can save someone else a similar headache. I am using a conda environment when getting this error and it was solved by starting a new environment where I made sure that every package was being taken from the conda-forge channel (except for some libraries only available from the esri channel. |
I am seeing the same behaviour with conda 4.9.2 on Windows 10 trying to get geopandas up and running. Creating an environment and forcing only conda-forge packages. As best as I can tell everything is "correctly" coming from conda-forge. various versions are: |
@janhurst can you post the output of |
@jorisvandenbossche I'm getting the same error with the same conda and packages versions as @janhurst . I have some non conda-forge packages which i believe have nothing to do with geopandas. I'm posting the exact error i'm getting and my conda list output.
|
I noticed that the environment seems to work ok when i try to read a file in a notebook when using jupyter lab, yet the same notebook will throw the error I posted when running inside vscode. @jjohnarios do you happen to be using vscode? I dug a bit further. If I start vscode from a Powershell prompt (eg However, If I first activate the "geo" environment (eg I'm pretty confused now :) ... I don't know if this is a vscode issue, or perhaps something screwy in conda and its Powershell integration etc. @jorisvandenbossche for reference my conda list output is attached, as well as the environment yaml. |
@janhurst I've tried vscode and import works fine. I was using jupyter notebook and spyder and the error kept occurring there. I made sure i was using the same kernel and environment. Consequently, i realised that : |
I have the exact same issue, where It should also be noted that this is only for an environment I have created using Python 3.6, however, if I create an environment with Python 3.8 I do not have this issue. @jorisvandenbossche any ideas what may be causing this issue? |
@jorisvandenbossche do you have an idea how to debug this? I don't have access to a Windows machine at the moment, so the only possibility would be to create a CI workflow that would reproduce this. But for that, we would require the exact steps to reproduce this issue. Maybe placing |
I also don't have a Windows machine readily available. @janhurst @jjohnarios @andrewnell can you try if setting the following environment variable before importing geopandas/fiona inside the notebook / spyder helps? It should be the equivalent of:
so in terms of python code you can run in the notebook before importing geopandas/fiona, I think it would be something like
|
I've tried that but doesn't seem to work. Same error message. |
@jjohnarios OK, thanks for trying! Aonther question: you posted above your |
Also didn't see any difference setting GDAL_DATA. |
Another question (sorry for the many questions, but that's at the moment the only way for me to help debug it): do you only see it with the latest fiona 1.8.18? If you explicitly install fiona 1.8.17 in the environment, do you still see the error? Or if you explicitly ask for an older GDAL? (you both seem to have 3.2.2 installed, which is relatively new (which might explain that it's a new issue)) Do you also have the error when creating the environment with |
I tried to follow the getting started linked as exactly as possible. The only addition is that the ipykernel package is needed to actually switch to the kernel in vscode's notebook environment. Unfortunately I am still seeing the same behaviour - starting vscode from the base environment and trying to select the geo_env kernel results in the For reference the conda list output for the environment as described in the geopandas getting started plus |
Missed this, its late here but I'll give this a shot in the next day or two. |
Hey, there, Having the same issue as @janhurst.
Tried with fiona 1.8.17 and 1.8.16 and still no success (same error)
Tried that too and still no success. Here is my current conda list output: Just reforcing: on anaconda prompt the import occurs successfully, on vscode it does not work. |
Probably a different bug, but I'm seeing a similar circular import error when I import geopandas (fiona 1.18.18, geopandas 0.9.0, gdal 3.2.2, installed with pip and brew on macos). The error goes away if I import fiona before geopandas. |
OK, still a mystery .. (and thanks all for the testing!) 1. Does this only happen with geopandas 0.9, or also with older versions (0.8)? 2. Something that changed in the latest geopandas version, is that we delayed the fiona import (geopandas/geopandas#1775), but which might have as concrete effect that the order of imports of the different geo-reletad packages changed.
3. Another possible "pattern" in the posted conda list outputs, is that you all have pyproj 3.0.1. Could someone try creating an environment but with an older version of pyproj? (3.0.0 or 2.6.1)? |
Hi, on MacOS using pip installs I also had this error when importing geopandas. Python 3.9.2. I changed pyproj from 3.0.1 to 3.0.0 and now the error goes away as long as I import fiona before geopandas. If I only import geopandas without fiona first, I still get the "Circular import" error message that others reported. |
Hi, I've stumbled across here too, and wanted to add my two cents. I'm running a script in VScode on windows 10, and only encounter the issue when executing via a Jupyter Interactive notebook session. I've encountered a lot of other dll issues with other libraries in the Jupyter Interactive window, that don't occur when running the script normally via the command line. There are a lot (of vaguely similar problems) spread across various github issues, but e.g. , vscode-jupyter and numpy As you mention in the above comment, activating an environment and calling |
Some general background context: conda has the concept of "environment activation" (which happens if you manually activate an environment in the console with Specifically for If those environment variables are not properly set, packages will have problems finding the correct binaries / libraries. And I suppose this is the problem that causes the issues you all encounter when using jupyter notebook or vscode. For Jupyter Notebook with using kernels based on conda environments, this is a known problem that this usage pattern doesn't automatically activate the environment when starting the kernel (jupyter/notebook#3588). One workaround is to install notebook in the environment you want to use, first activate the environment, and then start the notebook from that environment (this defeats the purpose of having multiple kernels a bit, though). Another solution is to use a tool like https://github.com/Anaconda-Platform/nb_conda_kernels that ensures kernels are activated properly. Finally, you could also prepend the appropriate For VSCode, I have less experience with this, but according to https://code.visualstudio.com/docs/python/environments#_select-and-activate-an-environment it seems that it should be possible to adjust your settings to ensure the environment is activated. Now, what I am less sure about is why it seems that there now new issues with this (given that multiple people posted here in the last weeks), and if something changed in fiona / gdal / the conda packages to make this problem worse / easier to run into. |
I've been trying to track this on my setup (macos, arm64, packages installed with brew and pip3, no conda) with little success. The first fiona import
But if I wrap a try/except around all the fiona imports in
The same doesn't happen if I import fiona directly, only from geopandas. I tried setting gdal datadir with There are some old mac specific bugs about the missing symbol in libspatialite, see e.g. Homebrew/homebrew-core#5161 maybe some recent change triggered them again. Still digging, I'll comment if I find anything new, if you have some test you'd like me to run let me know. |
I have a question for those of you who commented here and where "import fiona before geopandas" fixes the issue. To be able to investigate this issue, I would like to reproduce it (I have access to a Linux and Windows machine), but I currently don't know how. If some people that have this specific issue could describe their environment / how they installed geopandas and fiona / etc, that would be very helpful! |
@jorisvandenbossche I'm one of those but I only reproduced it on a macbook m1 (arm64) with packages installed from brew and pip, not sure if I can help. It's been a while since I looked into this, I should be able to run a quick test this evening to see if the issue is still there. |
@fargiolas if you would be able to check if you can still reproduce the issue, that would certainly be welcome! (let me know if you succeed, at which point I have a bunch of questions / things to test :) Are you using gitter? that might be a bit easier to chat about it) |
Not sure if this is useful but I encountered this exact circular import error today and was able to resolve it by pinning my
|
Prioritize `fiona` import to avoid circular dependencies (see [this coment](Toblerity/Fiona#944 (comment)) )
I am having this Circular imports issue only when i try to import fiona in jupyter notebook (everything works fine in my Pycharm). This workaround fixed my issue. @jorisvandenbossche |
We just hit the same circular import issue in our CI today, should this issue be reopened? |
A few months ago I had the issue on my CI tests on MacOS, but it disappeared again a few weeks later, and since yesterday the problem popped up on the windows tests... |
I think it's related to this new version of the conda feedstock released yesterday: conda-forge/fiona-feedstock#191 |
If only we had Windows wheels and didn't need to rely on conda... |
I had the same error messages as mentioned above on a fresh Python 3.10 install from Anaconda on Windows 10:
resulting in the following error message: ImportError: DLL load failed while importing ogrext: Kan opgegeven procedure niet vinden.
resulting in another error message: AttributeError: partially initialized module 'fiona' has no attribute 'loading' (most likely due to a circular import). The issue was solved by the following order of importing packages:
|
I also ran into that same issue and it drove me crazy for days.
This resulted in the below behaviors:
The packages were installed as follows using conda:
I tried installing GDAL from the GISInternals website as recommended on the Fiona PiPy page. But it did not solve the issue. Also in PyCharm/DataSpell/VSCode/Jupyter, the conda environment seems to be activated correctly. The os.environ["PATH"] and os.environ["GDAL_DATA"] are there and correct. None of the mentioned solutions mentioned in the thread above (or other forums) worked:
In the end what worked for me was the following:
I also noticed the following issue: If I import tensorflow before fiona.errors and geopandas then I get into the same error again "AttributeError: partially initialized module 'fiona' has no attribute '_loading' (most likely due to a circular import)" |
@mlnrt I am sorry to hear the troubles it caused, but thanks for the detailed write-up (it is a really annoying issue ..). |
@jorisvandenbossche Thank you for reach out. |
It could still be the case that importing fiona fails, and importing pyogrio works. But of course that's only useful to test in the env where fiona was actually failing to be imported. And if you don't have that installation at hand anymore, don't worry about it (not need to go back to your previous not-working setup!) |
+1 this worked for me. Thanks so much. On a python 3.8 conda forge environment built up a few weeks ago. If this resolves the issue, the issue has to be that the gdal path is messed up eh? Hard importing gdal in gives fiona/geopd the right path to point to but without it it fails? This is the second conda environment on this machine that needs gdal/fiona/geopandas (porting stuff from 3.7 to 3.8)... but there's only one path/one specific path each fiona/geopd can point to for all of them somehow? Any advice on updating that path (where and to what) would likely perma-solve the issue.... |
WIth fiona 1.8.15, the issue is fixed now in the conda-forge builds, only the Windows py3.8 build is still failing: conda-forge/fiona-feedstock#167
The failure is with the general error message of partially initialized module, so doesn't give much pointers (at least to me):
Originally posted by @jorisvandenbossche in #941 (comment)
The text was updated successfully, but these errors were encountered: