[there is no idle indexing node, waiting for retry...]

[moonSandra]

Hello,

I am currently facing an issue with my standalone Milvus instance, which unexpectedly went down, and I am unable to determine the cause. After restarting Milvus, I attempted to load my collection of 30 million vectors; however, the process terminates after approximately 10 minutes, and I am unsure of the reason.

The only relevant log message I have observed is as follows:

[2025/01/25 15:52:22.494 +00:00] [INFO] [datacoord/task_scheduler.go:258] ["there is no idle indexing node, waiting for retry..."]

I initially assumed that the indexing process needed to be restarted; however, the indexing has now been running for over two hours. I am considering allowing it more time to complete, but I would appreciate your guidance on whether this is expected behavior or if there are optimization strategies that could improve performance.

[xiaofan-luan]

1. check your cpu usage, see if the cpu is busy. (use pprof to check if index node takes most of your cpu)
2. please offer datacoord/indexcoord logs, we can help and check whether the log is working as expected
3. if you are using latest 2.5.3, 2.5.4 , try to use this webui and check the states of index task how much data is actually indexed https://milvus.io/docs/milvus-webui.md#Milvus-WebUI

[xiaofan-luan]

you should see all compaction and index build tasks in the task page.

[xiaofan-luan]

this is waiting to be loaded, not indexed?
Did you ever called collection load?

[moonSandra]

I waited 18 hours for the indexing operation to complete, and it finally finished shortly after my last reply. After that I tried collection loading. But the collection loading keeps stopping, and I can't figure out why. I tried loading it multiple times, but it discontinues after about 10-20 minutes.

[xiaofan-luan]

I waited 18 hours for the indexing operation to complete, and it finally finished shortly after my last reply. After that I tried collection loading. But the collection loading keeps stopping, and I can't figure out why. I tried loading it multiple times, but it discontinues after about 10-20 minutes.

is there a log for this?

[xiaofan-luan]

generally speaking, I don't think use one large standalone pod for huge amount of data is a good idea.
but just as index, it could be slow but should be able to succeed after a while. Unless you don't have enough memory or you hit into some other issue.

We will need logs to debug on this

[moonSandra]

Unfortunately, no. After repeatedly attempting this operation, I decided to remove the entire volume. I believe I had enough memory and CPU cores for handling just 70M records (at the beginning of inserting data from a total of 200M).

After that, I tried running Milvus with Docker Compose (installation guide) and modified the milvus.yaml file. Since I wasn’t familiar with the configuration variables, I experimented with adjusting the segment size and other parameters like timeouts.

Eventually, I got it working! Milvus ran for two days, successfully inserting, indexing, and loading about 170M records.

Now, I’m facing a new issue: the segment size doesn’t seem to have changed, as observed in the Web UI.

I would really appreciate any help in configuring milvus.yaml or any resources to better understand it.

Thank you so much for your responses and patience with someone who is new to Milvus.
milvus.txt

[xiaofan-luan]

First, download the milvus.yaml configuration file using the following command:

$ wget https://raw.githubusercontent.com/milvus-io/milvus/v2.5.4/configs/milvus.yaml
1

Modify the installation file

In docker-compose.yml, add a volumes section under milvus-standalone. Map your local milvus.yaml file path to the container configuration path /milvus/configs/milvus.yaml:

yaml
standalone:
container_name: milvus-standalone
image: milvusdb/milvus:v2.5.4
command: ["milvus", "run", "standalone"]
environment:
ETCD_ENDPOINTS: etcd:2379
MINIO_ADDRESS: minio:9000
volumes:

• /local/path/to/your/milvus.yaml:/milvus/configs/milvus.yaml # Map the local path to the container path
• ${DOCKER_VOLUME_DIRECTORY:-.}/volumes/milvus:/var/lib/milvus
  Start Milvus

After modifying both configuration and installation files, start Milvus with:

$ sudo docker compose up -d
Note that all parameter changes only take effect after Milvus restarts.

Just remember to change milvus.yaml, you need to mount milvus,.yaml into the right position of inside docker

[xiaofan-luan]

Also, I don't think you will need to tune any of these parameters, doesn't make a big difference.

You want to make sure:

1. don't do frequent flush, avoid generate too much segments.
2. use fast disk for etcd(SSD, better to be NVMe SSD or ebs)
3. use fast index algo, for instance IVFFLAT or IVFSQ8. since you only have one indexnode, HNSW or PQ index built might be slow

[moonSandra]

Thank you so much for your help.