[WIP] Automatically kill stuck PR tests and report back

[iarspider]

Split from #2418. For now, job list and criteria for killing builds are hardcoded. We can discuss how configurable do we want this job to be - for example, a dict mapping job name and filters on params to timeout.

[cmsbuild]

A new Pull Request was created by @iarspider for branch master.

@cmsbuild, @iarspider, @smuzaffar can you please review it and eventually sign? Thanks.
@antoniovilela, @mandrenguyen, @rappoccio, @sextonkennedy you are the release manager for this.
cms-bot commands are listed here

[cmsbuild]

cms-bot internal usage

[smuzaffar]

@iarspider , why localhost? this job runs on jenkins controller while the prs_commits.txt is available in cmssdt server

[iarspider]

• Proposed config structure:

• Add a separate job that runs on cmssdt to figure out commit_id based on upload_unique_id (and trigger commit status change from it) - https://cmssdt.cern.ch/jenkins/job/kill-stuck-pr-test/

• Post a PR comment that a test job was aborted due to all nodes being offline

• For jobs triggered by ib-run-pr-tests, pass a parameter that would mark them as valid targets for cleanup

• Pass full commit status name to jobs, instead of just prefix

[cmsbuild]

Pull request #2440 was updated.

[cmsbuild]

Pull request #2440 was updated.