tuxsuite: too many 500 error responses

[nickdesaulniers]

https://github.com/ClangBuiltLinux/continuous-integration2/actions/runs/5258852523/jobs/9530590012 under the tuxsuite dropdown has:

urllib3.exceptions.ResponseError: too many 500 error responses

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/requests/adapters.py", line 486, in send
    resp = conn.urlopen(
  File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 941, in urlopen
    return self.urlopen(
  File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 941, in urlopen
    return self.urlopen(
  File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 941, in urlopen
    return self.urlopen(
  [Previous line repeated 5 more times]
  File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 931, in urlopen
    retries = retries.increment(method, url, response=response, _pool=self)
  File "/usr/local/lib/python3.8/site-packages/urllib3/util/retry.py", line 515, in increment
    raise MaxRetryError(_pool, url, reason) from reason  # type: ignore[arg-type]
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='tuxapi.tuxsuite.com', port=443): Max retries exceeded with url: /v1/groups/clangbuiltlinux/projects/continuous-integration2/plans/2RA3pQQ7grenLOt36MX9uYbd3WB (Caused by ResponseError('too many 500 error responses'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/tuxsuite", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.8/site-packages/tuxsuite/cli/__init__.py", line 172, in main
    return plan_handlers[options.sub_command](options, extra_arguments, cfg)
  File "/usr/local/lib/python3.8/site-packages/tuxsuite/cli/plan.py", line 4[27](https://github.com/ClangBuiltLinux/continuous-integration2/actions/runs/5258852523/jobs/9530590012#step:4:28), in handle_submit
    result = wait_for_object(plan)
  File "/usr/local/lib/python3.8/site-packages/tuxsuite/cli/utils.py", line 78, in wait_for_object
    for state in build_object.watch():
  File "/usr/local/lib/python3.8/site-packages/tuxsuite/build.py", line 848, in watch
    plan = self.get_plan()
  File "/usr/local/lib/python3.8/site-packages/tuxsuite/build.py", line 750, in get_plan
    ret = get_request(
  File "/usr/local/lib/python3.8/site-packages/tuxsuite/build.py", line 51, in get_request
    response = tuxsuite.requests.get(url, headers=headers, params=params)
  File "/usr/local/lib/python3.8/site-packages/tuxsuite/requests.py", line 38, in wrapper
    response = f(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/tuxsuite/requests.py", line 55, in get
    return session.get(*args, timeout=timeout, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/requests/sessions.py", line 602, in get
    return self.request("GET", url, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python3.8/site-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/requests/adapters.py", line 510, in send
    raise RetryError(e, request=request)
requests.exceptions.RetryError: HTTPSConnectionPool(host='tuxapi.tuxsuite.com', port=443): Max retries exceeded with url: /v1/groups/clangbuiltlinux/projects/continuous-integration2/plans/2RA3pQQ7grenLOt[36](https://github.com/ClangBuiltLinux/continuous-integration2/actions/runs/5258852523/jobs/9530590012#step:4:37)MX9uYbd3WB (Caused by ResponseError('too many [50](https://github.com/ClangBuiltLinux/continuous-integration2/actions/runs/5258852523/jobs/9530590012#step:4:51)0 error responses'))

cc @vishalbhoj

[vishalbhoj]

This was caused by an AWS outage on June 13th.

[nickdesaulniers]

@nathanchance and I were discussing yesterday how to harden CI against this. Looking at our GH workflow files,

https://github.com/ClangBuiltLinux/continuous-integration2/blob/main/.github/workflows/4.14-clang-13.yml#L35

we do set if-no-files-found: error. The problem seems to be that tuxsuite produces an empty builds.json file when an error is thrown. We'd prefer to error out then and there.

Can tuxsuite be changed so that no builds.json is produced if an internal error is thrown?

[vishalbhoj]

@nickdesaulniers Let me check with the team on how best we can reflect the status back without a builds.json that is generated.