Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

potential tqdm issue #219

Open
1 task done
maxim-h opened this issue Jan 30, 2024 · 4 comments
Open
1 task done

potential tqdm issue #219

maxim-h opened this issue Jan 30, 2024 · 4 comments
Labels
bug Something isn't working

Comments

@maxim-h
Copy link

maxim-h commented Jan 30, 2024 •
edited
Loading

Failing to download dataset due to tqdm

Description of the bug

Installed pyega3 (from pip) and tried to download a dataset:

pyega3 -c 30 -cf CREDENTIALS_FILE fetch EGAD00001011283 --output-dir EGAD00001011283

But kept getting multiple errors even on re-try, so I cancelled the download.

Used versions (please complete the following information)

it is in the log 👇

  • Please, confirm you have tested your PyEGA3 installation (follow instructions):

To Reproduce

pyega3 -c 30 -cf CREDENTIALS_FILE fetch EGAD00001011283 --output-dir EGAD00001011283

Observed behaviour

Errors and no download

Expected behaviour

No errors and start of the download

Screenshots and error messages

pyega3_output.log 
[2024-01-30 13:50:04 +0100] pyEGA3 - EGA python client version 5.1.0 (https://github.com/EGA-archive/ega-download-client)
[2024-01-30 13:50:04 +0100] Parts of this software are derived from pyEGA (https://github.com/blachlylab/pyega) by James Blachly
[2024-01-30 13:50:04 +0100] Python version : 3.12.1
[2024-01-30 13:50:04 +0100] OS version : Linux #1 SMP Wed Jul 21 11:57:15 UTC 2021
[2024-01-30 13:50:04 +0100] Server URL: https://ega.ebi.ac.uk:8443/v2
[2024-01-30 13:50:04 +0100] Session-Id: 2772275406
[2024-01-30 13:50:17 +0100]
[2024-01-30 13:50:18 +0100] pyEGA3 - EGA python client version 5.1.0 (https://github.com/EGA-archive/ega-download-client)
[2024-01-30 13:50:18 +0100] Parts of this software are derived from pyEGA (https://github.com/blachlylab/pyega) by James Blachly
[2024-01-30 13:50:18 +0100] Python version : 3.12.1
[2024-01-30 13:50:18 +0100] OS version : Linux #1 SMP Wed Jul 21 11:57:15 UTC 2021
[2024-01-30 13:50:18 +0100] Server URL: https://ega.ebi.ac.uk:8443/v2
[2024-01-30 13:50:18 +0100] Session-Id: 2829523645
[2024-01-30 13:50:19 +0100]
[2024-01-30 13:50:19 +0100] Authentication success for user '[email protected]'
[2024-01-30 13:50:19 +0100] File Id: 'EGAF00008204178'(7061552690 bytes).
[2024-01-30 13:50:19 +0100] Total space : 77397.05 GiB
[2024-01-30 13:50:19 +0100] Used space : 69862.83 GiB
[2024-01-30 13:50:19 +0100] Free space : 7534.22 GiB
[2024-01-30 13:50:19 +0100] Download starting [using 30 connection(s), file size 7061552674 and chunk length 104857600]...
[2024-01-30 13:50:19 +0100] max() iterable argument is empty
Traceback (most recent call last):
  File "/g/zaugg/kholmato/.micromamba/envs/pyEGA3/lib/python3.12/site-packages/pyega3/libs/data_file.py", line 339, in _download_whole_file
    self.download_file(output_file, num_connections, max_slice_size)
  File "/g/zaugg/kholmato/.micromamba/envs/pyEGA3/lib/python3.12/site-packages/pyega3/libs/data_file.py", line 134, in download_file
    with tqdm(total=int(file_size), unit='B', unit_scale=True) as pbar:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/g/zaugg/kholmato/.micromamba/envs/pyEGA3/lib/python3.12/site-packages/tqdm/_tqdm.py", line 863, in __init__
    self.pos = self._get_free_pos(self)
               ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/g/zaugg/kholmato/.micromamba/envs/pyEGA3/lib/python3.12/site-packages/tqdm/_tqdm.py", line 470, in _get_free_pos
    return max(inst.pos for inst in cls._instances
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ValueError: max() iterable argument is empty
[2024-01-30 13:51:20 +0100] retry attempt 1
[2024-01-30 13:51:21 +0100] Download starting [using 30 connection(s), file size 7061552674 and chunk length 104857600]...
[2024-01-30 13:51:21 +0100] 'tqdm' object has no attribute 'pos'
Traceback (most recent call last):
  File "/g/zaugg/kholmato/.micromamba/envs/pyEGA3/lib/python3.12/site-packages/pyega3/libs/data_file.py", line 339, in _download_whole_file
    self.download_file(output_file, num_connections, max_slice_size)
  File "/g/zaugg/kholmato/.micromamba/envs/pyEGA3/lib/python3.12/site-packages/pyega3/libs/data_file.py", line 134, in download_file
    with tqdm(total=int(file_size), unit='B', unit_scale=True) as pbar:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/g/zaugg/kholmato/.micromamba/envs/pyEGA3/lib/python3.12/site-packages/tqdm/_tqdm.py", line 863, in __init__
    self.pos = self._get_free_pos(self)
               ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/g/zaugg/kholmato/.micromamba/envs/pyEGA3/lib/python3.12/site-packages/tqdm/_tqdm.py", line 470, in _get_free_pos
    return max(inst.pos for inst in cls._instances
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/g/zaugg/kholmato/.micromamba/envs/pyEGA3/lib/python3.12/site-packages/tqdm/_tqdm.py", line 470, in <genexpr>
    return max(inst.pos for inst in cls._instances
               ^^^^^^^^
AttributeError: 'tqdm' object has no attribute 'pos'

Additional context

I have solved the issue by upgrading the tqdm package via pip install tqdm --upgrade.
Not that it didn't complain about it:

pip install tqdm 
$ pip install tqdm --upgrade
Requirement already satisfied: tqdm in /[prefix]/.micromamba/envs/pyEGA3/lib/python3.12/site-packages (4.19.6)
Collecting tqdm
  Downloading tqdm-4.66.1-py3-none-any.whl.metadata (57 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 57.6/57.6 kB 3.4 MB/s eta 0:00:00
Downloading tqdm-4.66.1-py3-none-any.whl (78 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 78.3/78.3 kB 1.7 MB/s eta 0:00:00
Installing collected packages: tqdm
  Attempting uninstall: tqdm
    Found existing installation: tqdm 4.19.6
    Uninstalling tqdm-4.19.6:
      Successfully uninstalled tqdm-4.19.6
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following depend
ency conflicts.
pyega3 5.1.0 requires tqdm==4.19.6, but you have tqdm 4.66.1 which is incompatible.
Successfully installed tqdm-4.66.1

Despite the apparent error above after the upgrade the initial command worked and the download is currently working.

System information

I use micromamba with default channels just to get the base python installation.
Then I install everything with pip.

micromamba list 
List of packages in environment: "/[prefix]/.micromamba/envs/pyEGA3"

  Name              Version     Build            Channel
──────────────────────────────────────────────────────────
  _libgcc_mutex     0.1         main
  _openmp_mutex     5.1         1_gnu
  bzip2             1.0.8       h7b6447c_0
  ca-certificates   2023.12.12  h06a4308_0
  expat             2.5.0       h6a678d5_0
  ld_impl_linux-64  2.38        h1181459_1
  libffi            3.4.4       h6a678d5_0
  libgcc-ng         11.2.0      h1234567_1
  libgomp           11.2.0      h1234567_1
  libstdcxx-ng      11.2.0      h1234567_1
  libuuid           1.41.5      h5eee18b_0
  ncurses           6.4         h6a678d5_0
  openssl           3.0.12      h7f8727e_0
  pip               23.3.1      py312h06a4308_0
  python            3.12.1      h996f2a0_0
  readline          8.2         h5eee18b_0
  setuptools        68.2.2      py312h06a4308_0
  sqlite            3.41.2      h5eee18b_0
  tk                8.6.12      h1ccaba5_0
  tzdata            2023d       h04d1e81_0
  wheel             0.41.2      py312h06a4308_0
  xz                5.4.5       h5eee18b_0
  zlib              1.2.13      h5eee18b_0
pip freeze 
certifi==2023.11.17
charset-normalizer==2.0.12
htsget==0.2.5
humanize==4.9.0
idna==3.6
psutil==5.6.6
pyega3==5.1.0
requests==2.26.0
setuptools==68.2.2
six==1.16.0
tqdm==4.66.1
urllib3==1.26.7
wheel==0.41.2
@maxim-h maxim-h added the bug Something isn't working label Jan 30, 2024
@joshuamschmidt
Copy link

@maxim-h I ran into the same issue with python 3.12. Issue was resolved by downgrading to 3.6

@AndGob
Copy link

AndGob commented Apr 4, 2024

Thank you @joshuamschmidt and @maxim-h, I encountered the same problem in a conda enviroment. Downgrading python works also for me.
NOTE: All package need to be reisntalled after python downgrading.

@CIBRChenLab
Copy link

python=3.8 works for me, I also encountered error related to tqdm and request when using python 3.12

@sarangchafle
Copy link

I had a "ValueError: max() iterable argument is empty" while downloading a single file from a dataset. It pointed to an error with tqdm. I managed to resolve it by downgrading to python 3.8.15 in my virtual environment. Here is the error message for context.:

[2024-08-20 16:53:14 +0200] pyEGA3 - EGA python client version 5.2.0 (https://github.com/EGA-archive/ega-download-client) [2024-08-20 16:53:14 +0200] Parts of this software are derived from pyEGA (https://github.com/blachlylab/pyega) by James Blachly [2024-08-20 16:53:14 +0200] Python version : 3.12.3 [2024-08-20 16:53:14 +0200] OS version : Linux #40-Ubuntu SMP PREEMPT_DYNAMIC Fri Jul 5 10:34:03 UTC 2024 [2024-08-20 16:53:14 +0200] Server URL: https://ega.ebi.ac.uk:8443/v2 [2024-08-20 16:53:14 +0200] Session-Id: [REDACTED] [2024-08-20 16:53:17 +0200] [2024-08-20 16:53:17 +0200] Authentication success for user '[REDACTED]' [2024-08-20 16:53:18 +0200] File Id: 'EGAF00007808080'(29739770049 bytes). [2024-08-20 16:53:18 +0200] Total space : 96.68 GiB [2024-08-20 16:53:18 +0200] Used space : 18.73 GiB [2024-08-20 16:53:18 +0200] Free space : 73.00 GiB [2024-08-20 16:53:18 +0200] Download starting [using 1 connection(s), file size 29739770033 and chunk length 104857600]... [2024-08-20 16:53:18 +0200] max() iterable argument is empty Traceback (most recent call last): File "/home/[REDACTED]/myenv/lib/python3.12/site-packages/pyega3/libs/data_file.py", line 346, in _download_whole_file self.download_file(output_file, num_connections, max_slice_size) File "/home/[REDACTED]/myenv/lib/python3.12/site-packages/pyega3/libs/data_file.py", line 138, in download_file with tqdm(total=int(file_size), unit='B', unit_scale=True) as pbar: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/[REDACTED]/myenv/lib/python3.12/site-packages/tqdm/_tqdm.py", line 863, in __init__ self.pos = self._get_free_pos(self) ^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/[REDACTED]/myenv/lib/python3.12/site-packages/tqdm/_tqdm.py", line 470, in _get_free_pos return max(inst.pos for inst in cls._instances ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ValueError: max() iterable argument is empty ^C Traceback (most recent call last): File "/home/[REDACTED]/myenv/lib/python3.12/site-packages/pyega3/libs/data_file.py", line 346, in _download_whole_file self.download_file(output_file, num_connections, max_slice_size) File "/home/[REDACTED]/myenv/lib/python3.12/site-packages/pyega3/libs/data_file.py", line 138, in download_file with tqdm(total=int(file_size), unit='B', unit_scale=True) as pbar: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/[REDACTED]/myenv/lib/python3.12/site-packages/tqdm/_tqdm.py", line 863, in __init__ self.pos = self._get_free_pos(self) ^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/[REDACTED]/myenv/lib/python3.12/site-packages/tqdm/_tqdm.py", line 470, in _get_free_pos return max(inst.pos for inst in cls._instances ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ValueError: max() iterable argument is empty

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@joshuamschmidt @maxim-h @AndGob @CIBRChenLab @sarangchafle