Add taskcluster support to mozdownload

[whimboo]

Given several problems we face with querying archive.mozilla.org for builds, and the fact that the build system will be transitioned over to taskcluster and its index, we should add a taskcluster module which let us turn on support for taskcluster step by step for various type of builds.

I think what we have to get started with are the latest nightly builds on various branches:
https://bugzilla.mozilla.org/show_bug.cgi?id=703559#c60

Access will happen through the taskcluster index. Full documentation can be found here:
http://docs.taskcluster.net/services/index/

We could also have a look at mozregression given that it has that support already integrated.

[parkouss]

Here is an example to get a build url of a firefox 64 bits linux build:

import re
import taskcluster

re_firefox_linux64 = re.compile(r"firefox.*linux-x86_64\.tar.bz2$")

index = taskcluster.client.Index()
queue = taskcluster.Queue()

route = "gecko.v2.mozilla-central.nightly.2016.01.04.latest.firefox.linux64-opt"
task_id = index.findTask(route)['taskId']

artifacts = queue.listLatestArtifacts(task_id)["artifacts"]

for artifact in artifacts:
    if re_firefox_linux64.search(artifact['name']):
        build_url = queue.buildUrl('getLatestArtifact', task_id,
                                   artifact['name'])
        print(build_url)

In mozregression we do things a bit differently, checking for runs of a task, using different routes (revision based) in combination with pushlog api and allowing private artifact accesses, but the logic is similar. Here is the relevant mozregression code: https://github.com/mozilla/mozregression/blob/68df78fe694beb6e9dc9abe848bf2c7f53a1f1db/mozregression/fetch_build_info.py#L88-L172.

[whimboo]

Thanks Julien! So with mozdownload we have the following data for downloading a build:

• Release/Candidate: latest, version
• Daily: latest, date, buildid, (revision via Treeherder soon) + build number
• Tinderbox build: latest, date vs. timestamp, (revision via Treeherder soon) + debug flag, build number
• Tryscraper: revision + debug flag

To all of them variations of application, branch, locale, and platform will exist. Treeherder for Tinderbox is needed for the beta, release and esr branches given that there are no builds in the taskcluster index yet.

So the example above looks to be trivial and might not be too hard to get implemented. Lets see if I can do that the next weeks, or find a core contributor who is interested in doing that.

[Motwani]

Hi @whimboo I am interested in this issue. I have read up on taskcluster and confident I can solve this

[whimboo]

That sounds good. If you want, you can take it! One thing I want to note is that we might want to refactor mozdownload first so we can have the three possible download solutions as independent packages. Those are:

• Scrapers (archive.mozilla.org)
• Treeherder (by revision)
• Taskcluster

The Taskcluster method we might be able to use for everything nowadays. But at some point in the past we have to fallback to the scraper method because builds won't be available via the Taskcluster index.

[EricRahm]

This is more important now that we've stopped doing Linux buildbot builds. I looked into fixing things, but it boils down to a few tweaks in treeherder.py and then completely replacing the Scrapers with something that doesn't depend on archive.m.o

[EricRahm]

glandium pointed out that the code behind mach artifact install might be a useful starting point, there's also code in mozregression that could probably help.

[whimboo]

Lets wait for a response from Releng on bug 1344936 before making a decision what could be done with mozdownload, and especially when.

[armenzg]

<&jonasfj> I don't there is a tool, but:
2:11 PM an HTTPS client with retries, following redirects, supporting content-encoding: gzip, should be able to just fetch it from index.taskcluster.net/v1/task//artifacts/
2:13 PM docs: https://docs.taskcluster.net/reference/core/taskcluster-index/references/api#findArtifactFromTask
2:21 PM G<&garndt> armenzg: this is the latest artifacts indexed for win64 opt on inbound: https://tools.taskcluster.net/index/artifacts/#gecko.v2.mozilla-inbound.latest.firefox/gecko.v2.mozilla-inbound.latest.firefox.win64-opt
2:21 PM so for the latest installer for win64 opt on inbound: https://index.taskcluster.net/v1/task/gecko.v2.mozilla-inbound.latest.firefox.win64-opt/artifacts/public/build/firefox-56.0a1.en-US.win64.installer.exe

Also for reference:
taskcluster/taskcluster-rfcs#56

[davehunt]

I recently learned of fuzzfetch from @tysmith, developed and used by the fuzzing team.

[whimboo]

I actually stumbled about this posting:
https://groups.google.com/g/mozilla.dev.platform/c/0OV_M3b-cMM/m/kkknfFdECgAJ

As it lays out the Taskcluster builds will disappear after a year and as such wont' be accessible anymore. That means we would have to keep the scraping feature as fallback so that old binaries (>1 year) can still be downloaded.