[Bug]: Milvus start failed when trying to remove file and directory

[chyezh]

Is there an existing issue for this?

• I have searched the existing issues

Environment

- Milvus version: master-3a3404658e849bd275cb35abeddfb008aba5c183
- Deployment mode(standalone or cluster):
- MQ type(rocksmq, pulsar or kafka):    
- SDK version(e.g. pymilvus v2.0.0rc2):
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

[2024/10/30 18:04:12.891 +08:00] [WARN] [segments/cgo_util.go:86] ["CStatus returns err"] [error="boost::filesystem::file_size: No such file or directory [system:2]: \"/var/lib/milvus/data/indexnode/text_log/453580601963416497/0/453580601963416496/103/c9dbb3c27f2b40debfcaafc2e33ef45e.store\""] [extra="get local used size failed"]
[2024/10/30 18:04:12.891 +08:00] [WARN] [querynodev2/server.go:318] ["get local used size failed"] [error="boost::filesystem::file_size: No such file or directory [system:2]: \"/var/lib/milvus/data/indexnode/text_log/453580601963416497/0/453580601963416496/103/c9dbb3c27f2b40debfcaafc2e33ef45e.store\""]
[2024/10/30 18:04:12.891 +08:00] [ERROR] [querynode/service.go:144] ["QueryNode init error: "] [error="boost::filesystem::file_size: No such file or directory [system:2]: \"/var/lib/milvus/data/indexnode/text_log/453580601963416497/0/453580601963416496/103/c9dbb3c27f2b40debfcaafc2e33ef45e.store\""] [stack="github.com/milvus-io/milvus/internal/distributed/querynode.(*Server).init\n\t/home/chyezh/repository/chyezh/milvus/internal/distributed/querynode/service.go:144\ngithub.com/milvus-io/milvus/internal/distributed/querynode.(*Server).Run\n\t/home/chyezh/repository/chyezh/milvus/internal/distributed/querynode/service.go:222\ngithub.com/milvus-io/milvus/cmd/components.(*QueryNode).Run\n\t/home/chyezh/repository/chyezh/milvus/cmd/components/query_node.go:59\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/home/chyezh/repository/chyezh/milvus/cmd/roles/roles.go:129"]
[2024/10/30 18:04:12.891 +08:00] [ERROR] [components/query_node.go:60] ["QueryNode starts error"] [error="boost::filesystem::file_size: No such file or directory [system:2]: \"/var/lib/milvus/data/indexnode/text_log/453580601963416497/0/453580601963416496/103/c9dbb3c27f2b40debfcaafc2e33ef45e.store\""] [stack="github.com/milvus-io/milvus/cmd/components.(*QueryNode).Run\n\t/home/chyezh/repository/chyezh/milvus/cmd/components/query_node.go:60\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/home/chy
ezh/repository/chyezh/milvus/cmd/roles/roles.go:129"]

panic: boost::filesystem::file_size: No such file or directory [system:2]: "/var/lib/milvus/data/indexnode/text_log/453580601963416497/0/453580601963416496/103/c9dbb3c27f2b40debfcaafc2e33ef45e.store"

Expected Behavior

No panic should happen.

Steps To Reproduce

No response

Milvus Log

No response

Anything else?

No response

[chyezh]

Milvus standalone may be crash at startup if the /var/lib/milvus directory is not empty.

IndexNode or other component may clear the file at the directory,
but some component try to fetch the size of the directory.

the component fetch the size of directory may report failure if the file is removed, so startup failure happens.

[yanliang567]

/assign @chyezh
/unassign