Storage discovery: handle ResourceNotReady

[defo89]

Describe the bug
While server is in Discovery phase and is restarting or booting, Storage resource may not be ready to respond to request. Maybe this can be handled somehow?

2024-11-06T10:48:35.544570402Z 2024-11-06T10:48:35Z	ERROR	Reconciler error	{"controller": "server", "controllerGroup": "metal.ironcore.dev", "controllerKind": "Server", "Server": {"name":"node009-pod001-system-0"}, "namespace": "", "name": "node009-pod001-system-0", "reconcileID": "a313bec7-2675-419c-83ae-46b9bbac1bfa", "error": "failed to ensure server state transition: failed to get storages for Server: failed to retrieve some items: [{\"link\":\"/redfish/v1/Systems/1/Storage/\",\"error\":\"400: {\\\"error\\\":{\\\"code\\\":\\\"iLO.0.10.ExtendedInfo\\\",\\\"message\\\":\\\"See @Message.ExtendedInfo for more information.\\\",\\\"@Message.ExtendedInfo\\\":[{\\\"MessageArgs\\\":[\\\"5, (82,00,00)\\\"],\\\"MessageId\\\":\\\"iLO.2.27.ResourceNotReadyRetry\\\"}]}}\"}]"}
2024-11-06T10:48:35.544614080Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler
2024-11-06T10:48:35.544621554Z 	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:316
2024-11-06T10:48:35.544627533Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem
2024-11-06T10:48:35.544631934Z 	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:263
2024-11-06T10:48:35.544657010Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2
2024-11-06T10:48:35.544661277Z 	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:224

[defo89]

This particular error happened with

  manufacturer: HPE
  model: ProLiant DL380 Gen11

Will observe how it look on other servers.

[afritzler]

Seems like a similar issue we are seeing here: #170

[stefanhipfel]

Seems like a similar issue we are seeing here: #170

I use polling now to wait for redfish resources