[Bug]: [Nightly] Load occasionally failed reporting "wait for loading collection timeout"

[binbinlv]

Is there an existing issue for this?

• I have searched the existing issues

Environment

- Milvus version: 2.5 latest
- Deployment mode(standalone or cluster):cluster
- MQ type(rocksmq, pulsar or kafka):    kafka
- SDK version(e.g. pymilvus v2.0.0rc2): 2.5 latest
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

Load occasionally failed reporting "wait for loading collection timeout"

-02-12T22:22:10Z {container="step-test"} [2025-02-12 20:23:40 - DEBUG - ci_test]: (api_request)  : [Collection.load] args: [None, 1, 180], kwargs: {} (api_request.py:52)
2025-02-12T22:22:10Z {container="step-test"} [2025-02-12 20:26:40 - ERROR - ci_test]: Traceback (most recent call last):
2025-02-12T22:22:10Z {container="step-test"} File "/milvus/tests/python_client/utils/api_request.py", line 22, in inner_wrapper
2025-02-12T22:22:10Z {container="step-test"} res = func(*args, **_kwargs)
2025-02-12T22:22:10Z {container="step-test"} File "/milvus/tests/python_client/utils/api_request.py", line 53, in api_request
2025-02-12T22:22:10Z {container="step-test"} return func(*arg, **kwargs)
2025-02-12T22:22:10Z {container="step-test"} File "/usr/local/lib/python3.10/site-packages/pymilvus/orm/collection.py", line 427, in load
2025-02-12T22:22:10Z {container="step-test"} conn.load_collection(
2025-02-12T22:22:10Z {container="step-test"} File "/usr/local/lib/python3.10/site-packages/pymilvus/decorators.py", line 141, in handler
2025-02-12T22:22:10Z {container="step-test"} raise e from e
2025-02-12T22:22:10Z {container="step-test"} File "/usr/local/lib/python3.10/site-packages/pymilvus/decorators.py", line 137, in handler
2025-02-12T22:22:10Z {container="step-test"} return func(*args, **kwargs)
2025-02-12T22:22:10Z {container="step-test"} File "/usr/local/lib/python3.10/site-packages/pymilvus/decorators.py", line 176, in handler
2025-02-12T22:22:10Z {container="step-test"} return func(self, *args, **kwargs)
2025-02-12T22:22:10Z {container="step-test"} File "/usr/local/lib/python3.10/site-packages/pymilvus/decorators.py", line 107, in handler
2025-02-12T22:22:10Z {container="step-test"} raise MilvusException(
2025-02-12T22:22:10Z {container="step-test"} pymilvus.exceptions.MilvusException: <MilvusException: (code=1, message=Retry timeout: 180s, message=wait for loading collection timeout, collection: delete_cEPnAdbj)>
2025-02-12T22:22:10Z {container="step-test"} (api_request.py:35)
2025-02-12T22:22:10Z {container="step-test"} [2025-02-12 20:26:40 - ERROR - ci_test]: (api_response) : <MilvusException: (code=1, message=Retry timeout: 180s, message=wait for loading collection timeout, collection: delete_cEPnAdbj)> (api_request.py:36)

Expected Behavior

Load successfully

Steps To Reproduce

    @pytest.mark.tags(CaseLabel.L1)
    def test_delete_growing_data_channel_delete(self):
        """
        target: test delete entities from growing segment, and channel deleteMsg
        method: 1.create collection
                2.load collection
                3.insert data and delete ids
                4.query deleted ids
        expected: No query result
        """
        # create collection
        collection_w = self.init_collection_wrap(name=cf.gen_unique_str(prefix))
        # load collection and the queryNode watch the insertChannel
        collection_w.create_index(ct.default_float_vec_field_name, index_params=ct.default_flat_index)
        collection_w.load()
        # insert data
        df = cf.gen_default_dataframe_data(tmp_nb)
        collection_w.insert(df)
        # delete id 0
        del_res = collection_w.delete(tmp_expr)[0]
        assert del_res.delete_count == 1
        # query id 0
        collection_w.query(tmp_expr, check_task=CheckTasks.check_query_empty)

Milvus Log

artifacts-milvus-distributed-kafka-mdk-25-65-py-n-65-e2e-logs.tar.gz

Anything else?

collection: delete_cEPnAdbj
timeline: -02-12T22:22:10Z

test log:
https://jenkins.milvus.io:18080/blue/rest/organizations/jenkins/pipelines/Milvus%20Nightly%20CI(new)/branches/2.5/runs/65/nodes/134/steps/169/log/?start=0

[xiaofan-luan]

/assign @aoiasd
please help on debugging this