Error messages for bad URLs

[chvillanuevap]

One of the packages in my profile:

extends:
 - file: config.yaml

parameters:
  platform: Darwin
  fortran: false
  PATH: /usr/bin:/bin:/usr/sbin:/sbin
  PROLOGUE: |
    export MACOSX_DEPLOYMENT_TARGET=$(sw_vers -productVersion | sed -E "s/([0-9]+\.[0-9]+).*/\1/")

packages:
  launcher:
  blas:
    use: host-osx-framework-accelerate
  lapack:
    use: host-osx-framework-accelerate
  mpi:
    use: mpich
  python:
    link: shared
  doxygen:
  eigen:
  armadillo:

has a dependency on gdbm. When I build it, I get the following error:

engr2-19-152-dhcp [hashstack] ../hashdist/bin/hit build -j 4
Directory ba created.
Directory bld created.
Directory src created.
Directory db created.
Directory cache created.
Directory gcroots created.
Default configuration file /Users/chvillanuevap/.hashdist/config.yaml written.
[blas] Building blas/orqin7luuh5d, follow log with:
[blas]   tail -f /Users/chvillanuevap/.hashdist/tmp/blas-orqin7luuh5d/build.log
Downloading http://downloads.sourceforge.net/project/expat/expat/2.1.0/expat-2.1.0.tar.gz...
[=========================] 100.0% (0.5MB of 0.5MB) 0.183MB/s ETA 0s  
[expat] Building expat/n3nuaff47ht5, follow log with:
[expat]   tail -f /Users/chvillanuevap/.hashdist/tmp/expat-n3nuaff47ht5/build.log
Downloading http://releases.nixos.org/patchelf/patchelf-0.8/patchelf-0.8.tar.bz2...
[=========================] 100.0% (0.1MB of 0.1MB) 0.871MB/s ETA 0s  
[patchelf] Building patchelf/f5n57slqigu5, follow log with:
[patchelf]   tail -f /Users/chvillanuevap/.hashdist/tmp/patchelf-f5n57slqigu5/build.log
Downloading http://ftp.heanet.ie/mirrors/gnu/gdbm/gdbm-1.11.tar.gz...
[CRITICAL] Uncaught exception:
[CRITICAL] Traceback (most recent call last):
[CRITICAL]   File "/Users/chvillanuevap/Workspace/hashdist/hashdist/cli/main.py", line 202, in help_on_exceptions
[CRITICAL]     return func(*args, **kw)
[CRITICAL]   File "/Users/chvillanuevap/Workspace/hashdist/hashdist/cli/main.py", line 174, in command_line_entry_point
[CRITICAL]     retcode = args.subcommand_handler(ctx, args)
[CRITICAL]   File "/Users/chvillanuevap/Workspace/hashdist/hashdist/cli/frontend_cli.py", line 51, in run
[CRITICAL]     self.profile_builder_action()
[CRITICAL]   File "/Users/chvillanuevap/Workspace/hashdist/hashdist/cli/frontend_cli.py", line 108, in profile_builder_action
[CRITICAL]     self.args.k, self.args.debug)
[CRITICAL]   File "/Users/chvillanuevap/Workspace/hashdist/hashdist/spec/builder.py", line 147, in build
[CRITICAL]     self._package_specs[pkgname].fetch_sources(self.source_cache)
[CRITICAL]   File "/Users/chvillanuevap/Workspace/hashdist/hashdist/spec/package.py", line 61, in fetch_sources
[CRITICAL]     source_cache.fetch(source_clause['url'], source_clause['key'], self.name)
[CRITICAL]   File "/Users/chvillanuevap/Workspace/hashdist/hashdist/core/decorators.py", line 71, in f_retry
[CRITICAL]     return f(*args, **kwargs)
[CRITICAL]   File "/Users/chvillanuevap/Workspace/hashdist/hashdist/core/source_cache.py", line 345, in fetch
[CRITICAL]     handler.fetch(url, type, hash, repo_name)
[CRITICAL]   File "/Users/chvillanuevap/Workspace/hashdist/hashdist/core/source_cache.py", line 757, in fetch
[CRITICAL]     self.fetch_archive(url, type, hash)
[CRITICAL]   File "/Users/chvillanuevap/Workspace/hashdist/hashdist/core/source_cache.py", line 766, in fetch_archive
[CRITICAL]     return self._download_archive(url, type, expected_hash)
[CRITICAL]   File "/Users/chvillanuevap/Workspace/hashdist/hashdist/core/source_cache.py", line 770, in _download_archive
[CRITICAL]     temp_file, hash = self._download_and_hash(url, type)
[CRITICAL]   File "/Users/chvillanuevap/Workspace/hashdist/hashdist/core/source_cache.py", line 679, in _download_and_hash
[CRITICAL]     stream = urllib2.urlopen(url)
[CRITICAL]   File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 127, in urlopen
[CRITICAL]     return _opener.open(url, data, timeout)
[CRITICAL]   File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 404, in open
[CRITICAL]     response = self._open(req, data)
[CRITICAL]   File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 422, in _open
[CRITICAL]     '_open', req)
[CRITICAL]   File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 382, in _call_chain
[CRITICAL]     result = func(*args)
[CRITICAL]   File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 1214, in http_open
[CRITICAL]     return self.do_open(httplib.HTTPConnection, req)
[CRITICAL]   File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 1187, in do_open
[CRITICAL]     r = h.getresponse(buffering=True)
[CRITICAL]   File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 1045, in getresponse
[CRITICAL]     response.begin()
[CRITICAL]   File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 409, in begin
[CRITICAL]     version, status, reason = self._read_status()
[CRITICAL]   File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 373, in _read_status
[CRITICAL]     raise BadStatusLine(line)
[CRITICAL] BadStatusLine: ''
[CRITICAL] This exception has not been translated to a human-friendly error message,
[CRITICAL] please file an issue at https://github.com/hashdist/hashdist/issues pasting
[CRITICAL] this stack trace.

It seems the URL for gdbm does not exist. I got it to work by changing the source to:

sources:
- key: tar.gz:rwis6rhqlufrljff3fvhn6cs5ec5aun3
  url: ftp://ftp.gnu.org/gnu/gdbm/gdbm-1.11.tar.gz

Then, bzip2 fails in the same manner. So I had to change that.

My point is that the error message should be clear. The exception should say that the URL returned no data.

[chvillanuevap]

The bad URL was caused by my VPN's firewall; however, I still think the error message should indicate that the URL returned no data.

[certik]

For sure. These error messages are annoying me as well, they are not very clear.

[chvillanuevap]

The issue is that it raises a BadStatusLine exception, which can't be exclusively associated with a URL that returned no data, or a faulty internet connection.