Live-ISO test failing

[lentzi90]

Which jobs are failing?

Periodic E2E: https://github.com/metal3-io/baremetal-operator/actions/workflows/e2e-test-periodic-main.yml

Which tests are failing?

live-iso: https://github.com/metal3-io/baremetal-operator/actions/runs/12227994617/job/34105542268

Since when has it been failing?

Fist failure was 4th December

Jenkins link

No response

Reason for failure (if possible)

The BMH cannot be provisioned because the "image is not valid for use".

From Ironic logs:

2024-12-05 14:23:05.425 1 DEBUG ironic.common.images [None req-f1216348-13a7-4a46-bfcb-8de7d88b3fb1 - - - - - -] Image http://192.168.222.1/sysrescue-out.iso downloaded in 2.58 seconds. fetch_into /usr/lib/python3.9/site-packages/ironic/common/images.py:386
2024-12-05 14:23:05.430 1 DEBUG oslo_utils.imageutils.format_inspector [None req-f1216348-13a7-4a46-bfcb-8de7d88b3fb1 - - - - - -] Format inspector for vmdk does not match, excluding from consideration (Signature KDMV not found: b'3\xed\x90\x90') _process_chunk /usr/lib/python3.9/site-packages/oslo_utils/imageutils/format_inspector.py:1365
2024-12-05 14:23:05.435 1 DEBUG oslo_utils.imageutils.format_inspector [None req-f1216348-13a7-4a46-bfcb-8de7d88b3fb1 - - - - - -] Format inspector for vhdx does not match, excluding from consideration (Region signature not found at 30000) _process_chunk /usr/lib/python3.9/site-packages/oslo_utils/imageutils/format_inspector.py:1365
2024-12-05 14:23:05.437 1 ERROR ironic.common.images [None req-f1216348-13a7-4a46-bfcb-8de7d88b3fb1 - - - - - -] Security: The requested user image for the deployment node image cache failed to be able to be parsed by the image format checker: Multiple formats detected: iso,gpt: oslo_utils.imageutils.format_inspector.ImageFormatError: Multiple formats detected: iso,gpt
2024-12-05 14:23:05.512 1 ERROR ironic.conductor.utils [None req-f1216348-13a7-4a46-bfcb-8de7d88b3fb1 - - - - - -] Node f9ba65c4-987a-42b7-8965-1787c849a3f5 failed deploy step {'step': 'deploy', 'priority': 100, 'argsinfo': None, 'interface': 'deploy'}: The requested image is not valid for use.: ironic.common.exception.InvalidImage: The requested image is not valid for use.

Anything else we need to know?

We thought first that the issue was that we did not specify the image hash, but this has been ruled out in #2103.

Label(s) to be applied

/kind failing-test
One or more /area label. See https://github.com/metal3-io/baremetal-operator/labels for the list of labels.

[lentzi90]

/triage accepted

[tuminoid]

Added to CI tracker.

[tuminoid]

IIRC @Rozzii said this coincides with new ironic build on the same date.

[lentzi90]

I wonder if it could be related to this: https://bugs.launchpad.net/nova/+bug/2091114
We did a temporary workaround for it in CAPO.

[Rozzii]

I was expecting this originally: https://opendev.org/openstack/ironic/commit/669304bc0c6b2762c872b71297480c4b4ffdb554

[Rozzii]

Based on the quay artifacts, I would expect the root cause landed in ironic between Nov 6 and Dec 3.

[lentzi90]

I think you are correct that this came with oslo.utils.
The specific commit that introduced it in oslo.utils is here: openstack/oslo.utils@91af49b
Then there was some related changes that looks like they would be relevant to the live-iso use-case: openstack/oslo.utils@3d4ae16
I haven't figured out yet how we are supposed to "allow" multiple formats.

[Rozzii]

I think in the test system we supposed to have the safety check diasbled as we don't really care, but I have thought that enabling all the formats is the default.

[tuminoid]

BMO 0.8 virtualmedia works, where ironic is pinned to 26.0: #2111

[Rozzii]

I have created a bug report on Ironic side and continued working on this: https://bugs.launchpad.net/ironic/+bug/2091611

We could discuss reverting the pinning #2112 as there is a possibility to turn off the feature that initiates the problematic image format inspection, so that we could continue the testing of other Ironic features.

[tuminoid]

This is marked for 0.9. We have 2 rounds of workarounds implemented now, but the actual fix is pending. Should we close this as implemented in 0.9 and create another for 0.10 for the proper fix?

[lentzi90]

Sure we can do that or just change the milestone. Doesn't matter to me

[tuminoid]

It seems the revert is still failing, meaning main/0.9 is still broken. I guess whatever fix is going to be needs to be in 0.9 later on so let's actually keep 0.9 here.

[Rozzii]

/assign @Rozzii

[Rozzii]

Eventually in the Ironic community we have landed on this fix:
https://review.opendev.org/c/openstack/ironic/+/938363

[iurygregory]

Since the workaround in ironic is merged, should we https://review.opendev.org/c/openstack/ironic/+/938363, should we un-pin the ironic image? we need for #2229

[lentzi90]

Yes we should, but the tests are failing on the revert. I am debugging it at the moment

[dtantsur]

@lentzi90 it's not the same test that is failing, correct? I see the live ISO one passing, what fails is [It] provisions a BMH, applies detached and status annotations, then deprovisions.

[dtantsur]

2025-02-24 08:38:45.115 1 DEBUG ironic.drivers.modules.agent_base [-] deploy command status for node 155e7ecd-923d-4355-9456-2a398fddc33f on step {'step': 'write_image', 'priority': 80, 'argsinfo': None, 'interface': 'deploy'}: {'id': 'c6b68d87-b5cf-431e-bfe2-661eeb74b45f', 'command_name': 'execute_deploy_step', 'command_status': 'FAILED', 'command_error': {'type': 'CommandExecutionError', 'code': 500, 'message': 'Command execution failed', 'details': 'Unexpected error while running command.\nCommand: /opt/ironic-python-agent/bin/python3 -m oslo_concurrency.prlimit --as=2147483648 -- [\'env\', \'LC_ALL=C\', \'LANG=C\', \'qemu-img\', \'info\', \'/tmp/cirros-0.6.2-x86_64-disk.img\', \'--output=json\']\nExit code: 1\nStdout: \'\'\nStderr: "python3 -m oslo_concurrency.prlimit: failed to execute [\'env\', \'LC_ALL=C\', \'LANG=C\', \'qemu-img\', \'info\', \'/tmp/cirros-0.6.2-x86_64-disk.img\', \'--output=json\']: [Errno 2] No such file or directory\\n"'}, 'command_result': None} process_next_step /usr/lib/python3.9/site-packages/ironic/drivers/modules/agent_base.py:1094

This is Ironic logs but the error is from IPA, interesting.

[dtantsur]

Ramdisk log collection seems broken in that run :(

sed: -e expression #1, char 266: unknown option to `s'

EDIT: metal3-io/ironic-image#630

[dtantsur]

@lentzi90 I think I tracked this down to a bug in IPA: https://review.opendev.org/c/openstack/ironic-python-agent/+/942690

[lentzi90]

Thank you @dtantsur ! This has been driving me crazy!

[Rozzii]

Thank you @dtantsur !

[lentzi90]

I think I found another issue. 😦

While Ironic was pinned, we switched to kind because minikube was causing flakes. That obviously means we have not tested this with the latest ironic-image and it seems it does not work.

The difference is how Ironic is exposed to the BMHs. With minikube, we were running everything in the same network, but with kind, we have Ironic in a separate network. This means that we rely on the IRONIC_EXTERNAL_CALLBACK_URL for the callback and something about this has changed.
I think the issue is that we pass the base URL here. That would make IPA use the internal address instead of the external. I will try to test and fix this today.

Edit: Please disregard for now. I think this was my broken environment. It was not properly cleaned up before I tried the newer image.