OpenStack COS: Ignore openstack online runners

[cbartz]

Applicable spec: n/a

Overview

Ignore openstack online runners when extracting metrics.

Rationale

as it may be that the metrics have not yet been pulled.

We have seen a case in production where a runner had no status at all in Github, but the Openstack instance was still up. This led to a metrics extraction which removed the metrics storage, and on the next reconciliation the code tried to pull the metrics from the server, but threw a MetricsStorageError as the metrics could not be found:

2024-05-29 06:59:03 INFO unit.openstack-arm-large/21.juju-log server.go:325 Found 0 online and 0 offline openstack runners, 0 of the runners are busy	
2024-05-29 07:04:21 WARNING unit.openstack-arm-large/21.juju-log server.go:325 Unable to SSH into openstack-arm-large-21-fb2158b3a5b82820be9c1e79 with address 10.145.225.121	
2024-05-29 07:08:35 DEBUG unit.openstack-arm-large/21.juju-log server.go:325 Found openstack instances: [openstack.compute.v2.server.Server(id=ff598200-6eaf-4c49-a2ae-f74a60ecd738, name=openstack-arm-large-21-fb2158b3a5b82820be9c1e79, status=ACTIVE, ...
2024-05-29 07:09:20 DEBUG unit.openstack-arm-large/21.juju-log server.go:325 Extracting metrics from metrics storage for runner openstack-arm-large-21-fb2158b3a5b82820be9c1e79
2024-05-29 07:09:20 WARNING unit.openstack-arm-large/21.juju-log server.go:325 pre-job-metrics.json not found for runner openstack-arm-large-21-fb2158b3a5b82820be9c1e79.	
2024-05-29 07:09:20 DEBUG unit.openstack-arm-large/21.juju-log server.go:325 Cleaning metrics storage for runner openstack-arm-large-21-fb2158b3a5b82820be9c1e79
2024-05-29 07:21:52 INFO unit.openstack-arm-large/21.juju-log server.go:325 Pulling metrics and deleting server for OpenStack runner openstack-arm-large-21-fb2158b3a5b82820be9c1e79
2024-05-29 07:21:52 ERROR unit.openstack-arm-large/21.juju-log server.go:325 Failed to get shared metrics storage for runner openstack-arm-large-21-fb2158b3a5b82820be9c1e79, will not be able to issue all metrics.
2024-05-29 09:21:53.069	
  File "/var/lib/juju/agents/unit-openstack-arm-large-21/charm/src/openstack_cloud/openstack_manager.py", line 1352, in _pull_metrics
2024-05-29 09:21:53.069	
    storage = metrics_storage.get(instance_name)
2024-05-29 09:21:53.069	
  File "/var/lib/juju/agents/unit-openstack-arm-large-21/charm/src/metrics/storage.py", line 137, in get
2024-05-29 09:21:53.069	
    raise GetMetricsStorageError(f"Metrics storage for runner {runner_name} not found.")
2024-05-29 09:21:53.070	
errors.GetMetricsStorageError: Metrics storage for runner openstack-arm-large-21-fb2158b3a5b82820be9c1e79 not found.

Juju Events Changes

n/a

Module Changes

openstack_cloud.openstack_manager.OpenstackRunnerManager._issue_runner_metrics: Now lists the servers and ignores the servers=runners in ACTIVE mode for metrics extraction.

Library Changes

Checklist

• The charm style guide was applied
• The contributing guide was applied
• The changes are compliant with ISD054 - Managing Charm Complexity
• The documentation is generated using src-docs
• The documentation for charmhub is updated.
• The PR is tagged with appropriate label (urgent, trivial, complex)

[yanksyoon]

LGTM!