Failed Github Actions dependency download (succeeded on later retry)

[seanPhill]

When Github Actions was downloading dependencies and the source (at that time, https://gmplib.org/download/gmp) failed, it tried three times, and then tried the fallback source (Fetching gmp-6.1.2.tar.bz2 from https://bitcoincore.org/depends-sources) which also failed.

[seanPhill]

Others have complained about failures like this in bitcoin/bitcoin#22556 and suggested (in bitcoin/bitcoin#17234) having an extra download path available. The response was just that the FALLBACK_DOWNLOAD_PATH should be made to be more reliable, but Veil does not, of course, control bitcoincore.org, so the suggestion has been made that we should host a third 'depends' source at https://veil-project.com/depends-sources/. The issue was closed, because the specific files were later made to be present on the fallback, and the fallback should 'now' be highly reliable, and a third fallback has been made at https://drahtbot.space/depends_download_fallback/ however, this only contains currently required dependencies, and not necessarily dependencies for old Bitcoin versions, and therefore not dependencies for versions required by forks of old Bitcoin versions, and the same process that occasionally misses downloading a dependency to bitcoincore.org/depends-sources/ would all the more so, apply to veil-project.com/depends-sources, leaving extra, unproductive maintenance for the Veil team.

Regarding the failed download experienced in the beginning, gmp-6.1.2.tar.bz2, this could be updated to gmp-6.2.1.tar.bz2 by updating the package used, but on a subsequent Github Actions build forced, this time the original source did not fail to download it, and the Github Action succeeded.

The required download still exists and is available, but the source (and fallback) just happened to be non-responsive when originally tried. I think the appropriate response is to check Veil's dependencies and update versions where appropriate, and edit the original source if the URL has changed, or retry a bit later. It's a lot fiddlier to download it to Veil's website and let a specially-programmed third fallback get it there, and to continue to try four times to get it at known bad original source and fallback locations.

[seanPhill]

" the appropriate response is to check Veil's dependencies and update versions where appropriate, and edit the original source if the URL has changed, or retry a bit later."

[us77ipis]

The problem with the GitHub Actions pipeline failing was that gmplib.org seems to have had a problem resulting in the file gmp-6.1.2.tar.bz2 (temporarily) not being available. However, the bigger problem I see is that the file gmp-6.1.2.tar.bz2 wasn't available at the fallback url (bitcoincore.org) which seems to be a permanent unavailability. Thus, the redundancy that should be provided by the fallback url is not fulfilled anymore. That's why I suggested replacing the fallback by something hosted by the Veil Team (e.g. https://veil-project.com/depends-sources).

Setting that up should be quite easy:

1. In the cloned repo run make -C depends download to download all the current dependencies
2. Upload the downloaded files located under depends/sources to https://veil-project.com/depends-sources
3. Replace in the file depends/Makefile:10 the fallback url by https://veil-project.com/depends-sources

Independently of that, of course, we could think about updating some of the dependencies to newer versions, and try to update the original source urls for packages that are currently already using the bitcoincore.org fallback.

[seanPhill]

I suppose no-one else will object to not using bitcoincore.org's fallback more than me, so long as I'm the one to do this.

[seanPhill]

I'll make the PR to the website repo, and put some instructions in the Veil repo.

[ohcee]

Does this the issue address #568 too?