problems installing eomaps on Linux via pip

[langdonr621]

hi there,

1. trying to run the following code using the module:

from eomaps import Maps
import matplotlib.pyplot as plt
import pandas as pd

df = pd.read_csv('...csv')
m = Maps()
m.set_data(data=df, xcoord='...', ycoord='...', parameter='...', crs=4326)
m.set_shape.geod_circles(10000)
m.plot_map()

i was getting an exception thrown from the 'fiona' module:

Traceback (most recent call last):
  File "/opt/ws-py/eomaps/test.py", line 1, in <module>
    from eomaps import Maps, MapsGrid
  File "/opt/ws-py/eomaps/.env/lib/python3.10/site-packages/eomaps/__init__.py", line 11, in <module>
    from .eomaps import Maps, MapsGrid
  File "/opt/ws-py/eomaps/.env/lib/python3.10/site-packages/eomaps/eomaps.py", line 26, in <module>
    from cartopy import crs as ccrs
  File "/opt/ws-py/eomaps/.env/lib64/python3.10/site-packages/cartopy/__init__.py", line 108, in <module>
    import cartopy.feature  # noqa: F401  (flake8 = unused import)
  File "/opt/ws-py/eomaps/.env/lib64/python3.10/site-packages/cartopy/feature/__init__.py", line 18, in <module>
    import cartopy.io.shapereader as shapereader
  File "/opt/ws-py/eomaps/.env/lib64/python3.10/site-packages/cartopy/io/shapereader.py", line 43, in <module>
    import fiona
  File "/opt/ws-py/eomaps/.env/lib64/python3.10/site-packages/fiona/__init__.py", line 87, in <module>
    with fiona._loading.add_gdal_dll_directories():
AttributeError: partially initialized module 'fiona' has no attribute '_loading' (most likely due to a circular import). Did you mean: 'logging'?

replacing the lines:

import fiona._loading
with fiona._loading.add_gdal_dll_directories():

in .../.env/lib64/python3.10/site-packages/fiona/__init__.py --.env being the directory of a virtual env created inside the project where the code resides-- w/

import _loading
with _loading.add_gdal_dll_directories():

solved this problem.

2. running the code (a) terminates w/ no exception or stack trace, but (b) no map/plot window is shown.

adding

plt.show()

at the end causes the map's figure window to open which seems to solve the problem but i'm unsure of what consequences that work-around may cause later.

my environment is the following:

• Fedora 35, running GNU/Linux 5.15.6-200.fc35.x86_64
• Python 3.10.0

are these known problems, issues related to my environment, or something(s) i missed in installing and using the module?

TIA + cheers;

[raphaelquast]

hey, I've never had those problems... but it looks like they are connected to your venv and not to EOmaps...
people are experiencing similar difficulties independent from EOmaps: (Toblerity/Fiona#1030, Toblerity/Fiona#944 (comment))

How did you install all your dependencies?
I strongly suggest to start a fresh environment and use the ".yml" file provided in the documentation to setup all in one go using all packages from the conda-forge channel (this should make sure that the dependencies are all properly installed)

This is clearly related to GDAL which often creates problems if some of it's dependencies are not properly installed...

In your particular case it might be possible to circumvent the use of fiona by not adding coastlines (but in the long run I'd suggest a new venv) m.plot_map(coastlines=False)

[langdonr621]

hi,

thanks for your prompt reply!

i used pip to install the module and then installed the additional ones for WMS &al. support (i.e. owslib ...).

i had to do this multiple times due to missing packages in my Fedora installation that certain modules' wheels required --e.g. gdal-config mentioned in the fiona documentation. every time i did that i started w/ a clean virtual env. now i can pip install eomaps w/o problems in any clean/new virtual env. this leads me to believe that the problems i'm facing are unlikely to be related to my env.

i'd like to install a minimum of apps and packages so to that end i didn't opt for the conda-forge installation alternative. anyway, and just in case i missed the obvious, here's the list of modules present in my 'venv':

$ pip list
Package            Version
------------------ ---------
attrs              21.2.0
cairocffi          1.3.0
CairoSVG           2.5.2
Cartopy            0.20.1
certifi            2021.10.8
cffi               1.15.0
charset-normalizer 2.0.9
click              8.0.3
click-plugins      1.1.1
cligj              0.7.2
cssselect2         0.4.1
cycler             0.11.0
defusedxml         0.7.1
descartes          1.1.0
EOmaps             2.1.0
Fiona              1.8.20
fonttools          4.28.3
geopandas          0.10.2
idna               3.3
joblib             1.1.0
kiwisolver         1.3.2
mapclassify        2.4.3
matplotlib         3.5.0
munch              2.5.0
networkx           2.6.3
numpy              1.21.4
OWSLib             0.25.0
packaging          21.3
pandas             1.3.4
Pillow             8.4.0
pip                21.3.1
pycparser          2.21
pyepsg             0.4.0
pyparsing          3.0.6
pyproj             3.3.0
pyshp              2.1.3
python-dateutil    2.8.2
pytz               2021.3
PyYAML             6.0
requests           2.26.0
scikit-learn       1.0.1
scipy              1.7.3
setuptools         57.4.0
setuptools-scm     6.3.2
Shapely            1.8.0
six                1.16.0
threadpoolctl      3.0.0
tinycss2           1.1.1
tomli              1.2.2
urllib3            1.26.7
webencodings       0.5.1
wheel              0.36.2
xmltodict          0.12.0

additionally, i tried running the EOmaps examples and found that the work-around i mentioned earlier about replacing import fiona._loading w/ import _loading in fiona.__init__.py is not enough since that import style is used in few other places.

finally running the example WebMap services and layer-switching causes the following exception:

Traceback (most recent call last):
  File "/opt/ws-py/eomaps/test.py", line 436, in <module>
    ex6()
  File "/opt/ws-py/eomaps/test.py", line 403, in ex6
    wms1.add_layer.vv()
  File "/opt/ws-py/eomaps/.env/lib/python3.10/site-packages/eomaps/_containers.py", line 1391, in vv
    WMS.__doc__ = combdoc("Polarization: VV", type(self).__doc__)
  File "/opt/ws-py/eomaps/.env/lib/python3.10/site-packages/eomaps/_containers.py", line 24, in combdoc
    return "\n".join(dedent(i) for i in args)
  File "/opt/ws-py/eomaps/.env/lib/python3.10/site-packages/eomaps/_containers.py", line 24, in <genexpr>
    return "\n".join(dedent(i) for i in args)
  File "/usr/lib64/python3.10/textwrap.py", line 438, in dedent
    text = _whitespace_only_re.sub('', text)
TypeError: expected string or bytes-like object

this may be a different issue, or a Python 3.10 related one. don't know.

all the above IMO still do not properly explain the plt.show() needed to open the matplotlib window!

if there's anything i can test in my current environment to help find the cause(s) of the problem let me know. otherwise if you still believe it's an installation specific issue then close this report.

[raphaelquast]

I'm quite certain 99% of your problems are related to the way how you install the dependencies...

It's not just about making the installation of eomaps succeed... (it just means that the dependencies exist... not that they work properly). The problem is that a lot of dependencies (especially GDAL, GEOS, PYPROJ & co) are actually bindings to c-libraries which require a careful setup to avoid any mixup (even mixing channels with conda can lead to problems).
While it should in principle be possible to use pip, I personally find it very tedious and unreliable (e.g. the order of installation matters!) while using conda just works like a charm. (it will also only install the required packages and nothing more ... but it sorts out the correct dependencies for you such that all packages in your venv are satisfied...)

If you seek for a minimal environment, use miniconda !

I'm in the process of putting the package on conda-forge (to make the installation work out of the box using conda install -c conda-forge eomaps but this is a work-in-progress and will take some time since I have to wait for a response from the conda-forge team....

I can't really help you to get the Fedora + pip installation sorted out, but
the quickest (minimal) way to install that should work right away is the following:

1. install miniconda
2. open anaconda-prompt terminal and type conda install mamba
   (not required but it will speed up solving the dependencies a lot...
   if you don't use it, replace mamba with conda in step 4)
3. save yourself an eomaps.yml file with the content from here (install EOmaps from .yml)
4. run mamba env create -f "the path to the yml-file"
5. activate the env and you're good to go!

If you are willing to give it a try, I'd be interested if you still run into problems using the above approach!

... concerning the calls to plt.show()... for an ordinary-console you are right!
I've just released a new version (v2.1.1) that should no longer require an explicit call to plt.show()
thanks for mentioning!

[langdonr621]

hi Raphael,

thanks again for the prompt reply.

i'm not sure if miniconda is free + open-source software, which if it isn't i'm not interested in installing + using. however this is a moot point given that they only support up to Python 3.9 --see Linux Installers.

to verify my hunch about problems being related to Python version, i created a new clean project and virtual env and only installed fiona to try and run their basic example code given here. this caused the following exception:

Traceback (most recent call last):
  File "/opt/ws-py/fionatest/test.py", line 1, in <module>
    import fiona
  File "/opt/ws-py/fionatest/.env/lib64/python3.10/site-packages/fiona/__init__.py", line 86, in <module>
    from fiona.collection import BytesCollection, Collection
  File "/opt/ws-py/fionatest/.env/lib64/python3.10/site-packages/fiona/collection.py", line 11, in <module>
    from fiona.ogrext import Iterator, ItemsIterator, KeysIterator
ImportError: /opt/ws-py/fionatest/.env/lib64/python3.10/site-packages/fiona/ogrext.cpython-310-x86_64-linux-gnu.so: undefined symbol: _PyGen_Send

this looks like a Python 3.10 related issue which seems to be confirmed here.

glad to hear you sorted out the plt.show() issue!

i'll wait until newer versions of Fiona (and others) are released w/ support for Python 3.10 --hopefully before 3.11 is released ;-)-- to give the module another try.

thanks again for your time + pls. close as/when you see fit.

[raphaelquast]

hey, just to clarify a few things...

• conda is of course free and open source! https://github.com/conda/conda
  • ... so is mamba (a c++ reimplementation of conda to speed up things) https://github.com/mamba-org/mamba
• the python-version of the conda-installer only determines python in the base-environment (e.g. the version the package-manager is running in), you can of course put any python-version (incl. 3.10) in the venvs that you create on top of it!

... I've tried just out of curiosity to setup a fresh python 3.10 env using the yml file from the doc.
I don't get any problems and all works as expected.

[raphaelquast]

ok... I've just noticed that there is in fact an issue specific to the S1GBM wms service when using python 3.10...

I'll look into that #33 (other layers should work as expected... )

[langdonr621]

hi Raphael,

thanks for the pointers to Conda + Mamba. i'll try to learn more about those tools before deciding on installing and using them given that so far, w/ the exception of EOmaps, pip is working for me.

it should be obvious by now that:

1. there are indeed problems when using Python 3.10, and
2. installing EOmaps w/ pip on Fedora 35 does not guarantee a proper working installation.

don't know if the 2nd issue is a consequence of the first or not. my experiment w/ a Fiona only project seems to corroborate this but that's not a hard proof.

as a side-note, i upgraded EOmaps to 2.1.1 and now i don't get the plot window w/ or w/o adding the plt.show() call at the end of my code. reverting the 2 changes you made re. plt.ion/ioff/show, in eomaps.py, brings back the window again.

thanks for taking the time to look into this!

[raphaelquast]

Hey, thanks for your inputs! really appreciated!
I fixed the remaining wms service issue... (see #34) so now all is working as expected also in python 3.10 on my side...

If possible (and if you're willing to share your insights) I'd really like to come up with a solution to the plt.show() issue on unix systems before releasing another version...

If I understand correctly, you experience the following:

• in the initial version, a call to plt.show() was necessary to actually show the figure
• the current v2.1.1 implementation does not work and even invoking plt.show() manually does not show a figure

What matplotlib backend are you using? (on my side both Qt5Agg and TkAgg work as expected)

import matplotlib
matplotlib.get_backend()
>>> 'Qt5Agg'

[langdonr621]

hi Raphael,

your understanding is correct wrt. to the effect of plt.show() on both 2.1.0 and 2.1.1.

the only Matplotlib backend i had available was TkAgg. all other interactive alternatives mentioned here were not. i verified this by trying to force their use through a call to matplotlib.use(...) at the top of my code but that ended up raising exceptions.

after checking that i had the python3-qt5 5.15.6 package already installed (in my Fedora) i went ahead and did a pip install pyqt5 in the test project. at that point, the backend that gets used became QtAgg. also noticed that these additional ones were available for use(...): Qt5Agg, QtCairo, Qt5Cairo, and the original TkAgg.

unfortunately none of them show the plot's window when a pristine EOmaps 2.1.1 is used.

[raphaelquast]

OK... that's really odd...
To see if I can reproduce your issues, I've just tried fresh new environments on a pc running CentOS 7 using both python 3.7 and 3.10 with EOmaps v2.1.1 and all worked just as it should directly from the terminal... (I also tried both TkAgg and QtAgg matpltolib backends)

Since I can not reproduce the behaviour you're describing (neither on Windows nor on CenOS) I still think that the remaining issue might be connected to your python env...

my process of installing (with conda...) was:

1. install miniconda using bash "path to the .sh miniconda-installer"
2. setup env using the yml file from the documentation
   conda install mamba
mamba env create -f "path-to-the-yml-file"
conda activate eomaps
3. run a simple code like

   from eomaps import Maps
   m = Maps()
   m.plot_specs.crs = Maps.CRS.GOOGLE_MERCATOR
   m.add_wms.OpenStreetMap.add_layer.default()

[langdonr621]

hi Raphael,

i see you have this still open. in this case i'd like to add a comment about a point i raised earlier in that comment re. Fiona w/ Python 3.10

in that Fiona test project + code i mentioned which was raising an ImportError, i installed Fiona from its github master, less than an hour ago, w/

pip install -U git+https://github.com/Toblerity/Fiona.git#egg=fiona

which ended up w/...

...
Successfully installed fiona-2.0.dev0

the code now runs w/o exceptions and writes the output shapefile successfully.

[raphaelquast]

Well, that's good news!
How does this affect your experience with EOmaps? Is it now also running as expected or do you still have some problems?
(... as mentioned, on my side all works just fine in a fresh conda-environment on CentOS 7.)

... by the way... I've managed to put EOmaps on conda-forge in the meantime... so installation (incl. all dependencies) via conda is now as simple as: conda install -c conda-forge eomaps

[langdonr621]

hi Raphael,

even after installing the github master version of Fiona i still have the same problem using EOMaps to 2.1.1. specifically the Matplotlib plot window does not show up w/ or w/o the addition of plt.show() at the end of the client code.

[raphaelquast]

@langdonr621 since the release of EOmaps v3.0 the plot-window should pop up right after creating a Maps-object... if you're still interested it might be worth to give it another try 🙂

[raphaelquast]

I'll close this since there hasn't been any updates for quite some time (and the issues was last checked with v2.11 and we're now at v3.1)
Feel free to re-open if necessary!