Why is the step of reading the image so slow after a rebuild?

[rmast]

In Python-2.7 I use the standard Ubuntu 18.04 deb of python-gamera:
python-gamera_3.4.2+git20160808.1725654-2_amd64.deb

When I rebuild gamera with option nowx in Mint 20.2 it is reading images much slower. Same with gamera-4.

Has the deb been built with an optimization I don't know of or is 3.4.4 slower than 3.4.2 reading images?

The read-step is done from didjvu: 2 minute 45.

date;./didjvu encode ~/Afbeeldingen/outputbase2-000-raar-effect-onderste-regel-didjvu\ zonder\ tekst.tif -o jaarverslagraa5r.djvu;date
wo 17 nov 2021  7:21:20 CET
/home/robert/Afbeeldingen/outputbase2-000-raar-effect-onderste-regel-didjvu zonder tekst.tif:
- reading image
- converting to DjVu
- 0.010 bits/pixel; 113.343:1, 99.12% saved, 4122172 bytes in, 36369 bytes out
wo 17 nov 2021  7:24:05 CET

pipdeptree
argparse==1.2.1
ExifRead==2.3.2
gamera==3.4.4
gyp==0.1
image==1.5.33
  - django [required: Any, installed: 1.11.29]
    - pytz [required: Any, installed: 2021.3]
  - pillow [required: Any, installed: 6.2.1]
  - six [required: Any, installed: 1.16.0]
nose==1.3.7
numpy==1.16.6
olefile==0.46
pipdeptree==2.2.0
  - pip [required: >=6.0.0, installed: 20.3.4]
pyexiv2==1.2.1
Python==2.7.18
setuptools==44.1.1
wheel==0.37.0
wsgiref==0.1.2

[rmast]

I've reinstalled this deb on a fresh install of linuxmint-20.1-cinnamon-64bit.iso (see my other finding for the specific picture)

sudo apt-get install gdebi
sudo gdebi Downloads/python-gamera_3.4.2+git20160808.1725654-2_amd64.deb 
sudo apt install git
git clone https://github.com/jwilk/didjvu.git
cd didjvu
sudo apt-get install python-pil
sudo apt-get install djvulibre-bin
date;./didjvu encode ~/Pictures/outputbase2-000-raar-effect-onderste-regel-didjvu\ zonder\ tekst.tif -o jaarverslagraa5r.djvu;date
Wed 17 Nov 2021 07:14:59 AM CET
/home/oem/Pictures/outputbase2-000-raar-effect-onderste-regel-didjvu zonder tekst.tif:
- reading image
- converting to DjVu
- 0.010 bits/pixel; 113.066:1, 99.12% saved, 4122172 bytes in, 36458 bytes out
Wed 17 Nov 2021 07:15:08 AM CET

So only 9 seconds

cd didjvu
pipdeptree
argparse==1.2.1
gamera==3.4.2
olefile==0.46
Pillow==6.2.1
pipdeptree==2.2.0
  - pip [required: >=6.0.0, installed: 20.3.4]
pipreqs==0.3.0
  - docopt [required: Any, installed: 0.6.2]
  - yarg [required: Any, installed: 0.1.9]
    - requests [required: Any, installed: 2.26.0]
      - certifi [required: >=2017.4.17, installed: 2021.10.8]
      - chardet [required: >=3.0.2,<5, installed: 4.0.0]
      - idna [required: >=2.5,<3, installed: 2.10]
      - urllib3 [required: >=1.21.1,<1.27, installed: 1.26.7]
Python==2.7.18
setuptools==44.1.1
wheel==0.37.0
wsgiref==0.1.2

[rmast]

I'm now rebuilding the same gamera version as the deb on this new Mint 20.1 x64 image.

sudo add-apt-repository universe
sudo apt update
curl https://bootstrap.pypa.io/pip/2.7/get-pip.py --output get-pip.py
sudo python2 get-pip.py
pip -V
tail -n1 .bashrc:
export PATH=$PATH:/home/oem/.local/bin
pip install pipdeptree
sudo apt-get install --reinstall ca-certificates
git clone https://salsa.debian.org/python-team/packages/gamera.git
cd gamera
sudo apt-get install python-dev
git checkout 0794480a83ca8327706eb80f260d4b20cf6ae0aa

remove 'const' in the pyramid function above line 771 in include/vigra/imagecontainer.hxx
sudo apt-get install g++ libpng-dev libtiff-dev
python setup.py --nowx build

[rmast]

And the resulting time is long:

date;./didjvu encode ~/Pictures/outputbase2-000-raar-effect-onderste-regel-didjvu\ zonder\ tekst.tif -o jaarverslagraa5r.djvu;date
Wed 17 Nov 2021 01:54:19 PM CET
/home/oem/Pictures/outputbase2-000-raar-effect-onderste-regel-didjvu zonder tekst.tif:
- reading image
- converting to DjVu
- 0.010 bits/pixel; 113.346:1, 99.12% saved, 4122172 bytes in, 36368 bytes out
Wed 17 Nov 2021 01:57:37 PM CET

pipdeptree (same before and after sudo python setup.py --nowx install!)
argparse==1.2.1
docopt==0.6.2
gamera==3.4.2
olefile==0.46
Pillow==6.2.1
pipdeptree==2.2.0
  - pip [required: >=6.0.0, installed: 20.3.4]
Python==2.7.18
setuptools==44.1.1
wheel==0.37.0
wsgiref==0.1.2
yarg==0.1.9
  - requests [required: Any, installed: 2.26.0]
    - certifi [required: >=2017.4.17, installed: 2021.10.8]
    - chardet [required: >=3.0.2,<5, installed: 4.0.0]
    - idna [required: >=2.5,<3, installed: 2.10]
    - urllib3 [required: >=1.21.1,<1.27, installed: 1.26.7]

[rmast]

Finally, the commit in the debian repository that comes closest to the Python text when I compare the installed apt-package with the installed compiled version
diff /usr/local/lib/python2.7/dist-packages/gamera /usr/lib/python2.7/dist-packages/gamera

is this one:
73716f1f0b673814371aa5d4cea6670d90dcd329

But apart from the needed change in include/vigra/imagecontainer.hxx this commit still is too slow. Much slower than only could depend on the amount of processor-cores in my system.

But before I continue on this path I will try building with openmp-support.

[rmast]

With openmp support the same long time:

date;./didjvu encode ~/Pictures/outputbase2-000-raar-effect-onderste-regel-didjvu\ zonder\ tekst.tif -o jaarverslagraa5r.djvu;date
Wed 17 Nov 2021 05:22:34 PM CET
/home/oem/Pictures/outputbase2-000-raar-effect-onderste-regel-didjvu zonder tekst.tif:
- reading image
- converting to DjVu
- 0.010 bits/pixel; 113.677:1, 99.12% saved, 4122172 bytes in, 36262 bytes out
Wed 17 Nov 2021 05:26:01 PM CET

[rmast]

I don't know what the use is of

__init__.py(c)

in the gamera-submap, but when I remove them even the newer Gamera-4 version only takes seconds.

[rmast]

@cdalitz Even when I do a full build without gamera/__init__.py in the source tree the build runs fine. gamera/__init__.py causes a great lag, from 9 seconds to about 3 minutes runtime both in the 3.4 and 4 versions. Could you just drop gamera/__init__.py from the sourcetree?

[cdalitz]

This is very strange and I cannot reproduce your problem. gamer/init.py only contians two lines that do anything:

from . import __version__
__version__ = __version__.ver

Under what circumstances can this last 9 seconds?

[rmast]

Under what circumstances can this last 9 seconds?

See my second post in this issue for the exact replay, using the original package manager version of Ubuntu 18.04 for Gamera.

The used image comes from
wget https://github.com/jwilk/didjvu/files/7421026/outputbase2-000-raar-effect-onderste-regel-didjvu.zonder.tekst.zip

I've looked up the differences between the source of the package manager version and only found this file to be different, however if I remove it now it doesn't run, so there must be something else.

[rmast]

I replayed the scenario with Python3.9 and Python3.10 and there was no speed issue. There might be an issue with some reason for reading some Python-sourcefile(s) again and again in older Python-versions, but I'm not skilled enough to investigate that yet. Processmonitor isn't as advanced on Ubuntu as it is on Windows.