Openstack COS: Issue metrics even if runner creation fails

[cbartz]

Applicable spec: n/a

Overview

1. Output runner/reconciliation metrics even if runner creation fails
2. Increase server timeout to 5 minutes.
3. Also refactor the big reconcile method a bit.

Rationale

1. Otherwise there is no observability
2. 2 minutes may be too short
3. method was too complex (linter also complained)

Juju Events Changes

n/a

Module Changes

openstack_cloud.openstack_manager.OpenstackRunnerManager.reconcile has been refactored

Library Changes

Checklist

• The charm style guide was applied
• The contributing guide was applied
• The changes are compliant with ISD054 - Managing Charm Complexity
• The documentation is generated using src-docs
• The documentation for charmhub is updated.
• The PR is tagged with appropriate label (urgent, trivial, complex)

[amandahla]

Out of curiosity: when it fails by timeout, the server creation is aborted? Or could it fail by timeout but the server is still being created and at some point, is done and ready?

[cbartz]

I think it will not be cancelled, this is also the reason why we have a cleanup mechanism in those cases, but perhaps @yhaliaw can confirm.