What can cause data loss in S3? #36112

elpsycongroo · 2024-09-09T07:34:21Z

elpsycongroo
Sep 9, 2024

Today I find that some collections created before is gone:

And before that, the queryNode pod was restarted multiple times, with the following logs:

[2024/09/09 07:16:30.470 +00:00] [ERROR] [funcutil/parallel.go:88] [loadSegmentFunc] [error="At LoadSegment: Error in GetObjectSize[errcode:404, exception:, errmessage:No response body., params:params, bucket=bucket-name, object=file/index_files/451522410132278794/1/451522410132278172/451522410132278542/IVF_FLAT]"] [errorVerbose="At LoadSegment: Error in GetObjectSize[errcode:404, exception:, errmessage:No response body., params:params, bucket=bucket-name, object=file/index_files/451522410132278794/1/451522410132278172/451522410132278542/IVF_FLAT]\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/internal/querynodev2/segments.(*segmentLoader).Load.func4\n  | \t/go/src/github.com/milvus-io/milvus/internal/querynodev2/segments/segment_loader.go:708\n  | github.com/milvus-io/milvus/pkg/util/funcutil.ProcessFuncParallel.func3\n  | \t/go/src/github.com/milvus-io/milvus/pkg/util/funcutil/parallel.go:86\n  | runtime.goexit\n  | \t/usr/local/go/src/runtime/asm_amd64.s:1598\nWraps: (2) At LoadSegment\nWraps: (3) Error in GetObjectSize[errcode:404, exception:, errmessage:No response body., params:params, bucket=bucket-name, object=file/index_files/451522410132278794/1/451522410132278172/451522410132278542/IVF_FLAT]\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"] [idx=0] [stack="github.com/milvus-io/milvus/pkg/util/funcutil.ProcessFuncParallel.func3\n\t/go/src/github.com/milvus-io/milvus/pkg/util/funcutil/parallel.go:88"]
[2024/09/09 07:16:30.470 +00:00] [DEBUG] [funcutil/parallel.go:54] [loadSegmentFunc] [total=1] ["time cost"=12.381613ms]
[2024/09/09 07:16:30.470 +00:00] [WARN] [segments/segment_loader.go:734] ["failed to load some segments"] [traceID=1ec1809834139904f45f335dd18570eb] [collectionID=451522410132278171] [segmentType=Sealed] [requestSegments="[451522410132278542]"] [preparedSegments="[451522410132278542]"] [error="At LoadSegment: Error in GetObjectSize[errcode:404, exception:, errmessage:No response body., params:params, bucket=bucket-name, object=file/index_files/451522410132278794/1/451522410132278172/451522410132278542/IVF_FLAT]"] [errorVerbose="At LoadSegment: Error in GetObjectSize[errcode:404, exception:, errmessage:No response body., params:params, bucket=bucket-name, object=file/index_files/451522410132278794/1/451522410132278172/451522410132278542/IVF_FLAT]\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/internal/querynodev2/segments.(*segmentLoader).Load.func4\n  | \t/go/src/github.com/milvus-io/milvus/internal/querynodev2/segments/segment_loader.go:708\n  | github.com/milvus-io/milvus/pkg/util/funcutil.ProcessFuncParallel.func3\n  | \t/go/src/github.com/milvus-io/milvus/pkg/util/funcutil/parallel.go:86\n  | runtime.goexit\n  | \t/usr/local/go/src/runtime/asm_amd64.s:1598\nWraps: (2) At LoadSegment\nWraps: (3) Error in GetObjectSize[errcode:404, exception:, errmessage:No response body., params:params, bucket=bucket-name, object=file/index_files/451522410132278794/1/451522410132278172/451522410132278542/IVF_FLAT]\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"]
[2024/09/09 07:16:30.470 +00:00] [WARN] [segments/segment_loader.go:643] ["release new segment created due to load failure"] [traceID=1ec1809834139904f45f335dd18570eb] [collectionID=451522410132278171] [segmentType=Sealed] [requestSegments="[451522410132278542]"] [preparedSegments="[451522410132278542]"] [segmentID=451522410132278542] [error="At LoadSegment: Error in GetObjectSize[errcode:404, exception:, errmessage:No response body., params:params, bucket=bucket-name, object=file/index_files/451522410132278794/1/451522410132278172/451522410132278542/IVF_FLAT]"] [errorVerbose="At LoadSegment: Error in GetObjectSize[errcode:404, exception:, errmessage:No response body., params:params, bucket=bucket-name, object=file/index_files/451522410132278794/1/451522410132278172/451522410132278542/IVF_FLAT]\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/internal/querynodev2/segments.(*segmentLoader).Load.func4\n  | \t/go/src/github.com/milvus-io/milvus/internal/querynodev2/segments/segment_loader.go:708\n  | github.com/milvus-io/milvus/pkg/util/funcutil.ProcessFuncParallel.func3\n  | \t/go/src/github.com/milvus-io/milvus/pkg/util/funcutil/parallel.go:86\n  | runtime.goexit\n  | \t/usr/local/go/src/runtime/asm_amd64.s:1598\nWraps: (2) At LoadSegment\nWraps: (3) Error in GetObjectSize[errcode:404, exception:, errmessage:No response body., params:params, bucket=bucket-name, object=file/index_files/451522410132278794/1/451522410132278172/451522410132278542/IVF_FLAT]\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"]
[2024/09/09 07:16:30.470 +00:00] [INFO] [segments/segment.go:1482] ["delete segment from memory"] [collectionID=451522410132278171] [partitionID=451522410132278172] [segmentID=451522410132278542] [segmentType=Sealed] [insertCount=5000]

proxy log:

[2024/09/09 07:29:22.188 +00:00] [DEBUG] [proxy/impl.go:3474] ["Query received"] [traceID=355d2f946b7c58cbb1c075eacc54730f] [role=proxy] [db=default] [collection=newCollection] [partitions="[]"] [useDefaultConsistency=false] [expr="id > -9223372036854775807"] [OutputFields="[id,vector,$meta]"] [travel_timestamp=0] [guarantee_timestamp=0]
[2024/09/09 07:29:22.188 +00:00] [DEBUG] [proxy/impl.go:3500] [tr/Query] [traceID=355d2f946b7c58cbb1c075eacc54730f] [msg="query request enqueue"] [duration=108.566µs]
[2024/09/09 07:29:22.188 +00:00] [DEBUG] [proxy/impl.go:3502] ["Query enqueued"] [traceID=355d2f946b7c58cbb1c075eacc54730f] [role=proxy] [db=default] [collection=newCollection] [partitions="[]"] [useDefaultConsistency=false]
[2024/09/09 07:29:22.188 +00:00] [DEBUG] [proxy/task_query.go:289] ["Validate collectionName."] [traceID=355d2f946b7c58cbb1c075eacc54730f] [collectionName=newCollection] [partitionNames="[]"] [requestType=query]
[2024/09/09 07:29:22.188 +00:00] [DEBUG] [proxy/task_query.go:297] ["Get collection ID by name"] [traceID=355d2f946b7c58cbb1c075eacc54730f] [collectionName=newCollection] [partitionNames="[]"] [requestType=query] [collectionID=451522410132294244]
[2024/09/09 07:29:22.188 +00:00] [DEBUG] [proxy/task_query.go:318] ["Validate partition names."] [traceID=355d2f946b7c58cbb1c075eacc54730f] [collectionName=newCollection] [partitionNames="[]"] [requestType=query]
[2024/09/09 07:29:22.188 +00:00] [DEBUG] [proxy/task_query.go:242] ["translate output fields to field ids"] [traceID=355d2f946b7c58cbb1c075eacc54730f] [OutputFieldsID="[100,101,102,1]"] [requestType=query]
[2024/09/09 07:29:22.188 +00:00] [DEBUG] [proxy/task_query.go:439] ["Query PreExecute done."] [traceID=355d2f946b7c58cbb1c075eacc54730f] [collectionName=newCollection] [partitionNames="[]"] [requestType=query] [guarantee_ts=0] [mvcc_ts=0] [timeout_ts=452425672867184640]
[2024/09/09 07:29:22.189 +00:00] [INFO] [proxy/meta_cache.go:928] ["no shard cache for collection, try to get shard leaders from QueryCoord"] [traceID=355d2f946b7c58cbb1c075eacc54730f] [collectionName=newCollection] [collectionID=451522410132294244]
[2024/09/09 07:29:22.190 +00:00] [WARN] [proxy/lb_policy.go:203] ["failed to get shards"] [traceID=355d2f946b7c58cbb1c075eacc54730f] [error="segment lacks[segment=451522410132294501]: channel not available[channel=by-dev-rootcoord-dml_7_451522410132294244v0]"]
[2024/09/09 07:29:22.190 +00:00] [WARN] [proxy/task_query.go:462] ["fail to execute query"] [traceID=355d2f946b7c58cbb1c075eacc54730f] [collection=451522410132294244] [partitionIDs="[]"] [requestType=query] [error="segment lacks[segment=451522410132294501]: channel not available[channel=by-dev-rootcoord-dml_7_451522410132294244v0]"]
[2024/09/09 07:29:22.190 +00:00] [DEBUG] [proxy/task_query.go:463] ["tr/proxy execute query 451522410114063644"] [traceID=355d2f946b7c58cbb1c075eacc54730f] [msg=done] [duration=1.17516ms]
[2024/09/09 07:29:22.190 +00:00] [WARN] [proxy/task_scheduler.go:469] ["Failed to execute task: "] [traceID=355d2f946b7c58cbb1c075eacc54730f] [error="failed to query: segment lacks[segment=451522410132294501]: channel not available[channel=by-dev-rootcoord-dml_7_451522410132294244v0]"] [errorVerbose="failed to query: segment lacks[segment=451522410132294501]: channel not available[channel=by-dev-rootcoord-dml_7_451522410132294244v0]\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/internal/proxy.(*queryTask).Execute\n  | \t/go/src/github.com/milvus-io/milvus/internal/proxy/task_query.go:463\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).processTask\n  | \t/go/src/github.com/milvus-io/milvus/internal/proxy/task_scheduler.go:466\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).queryLoop.func1\n  | \t/go/src/github.com/milvus-io/milvus/internal/proxy/task_scheduler.go:545\n  | github.com/milvus-io/milvus/pkg/util/conc.(*Pool[...]).Submit.func1\n  | \t/go/src/github.com/milvus-io/milvus/pkg/util/conc/pool.go:81\n  | github.com/panjf2000/ants/v2.(*goWorker).run.func1\n  | \t/go/pkg/mod/github.com/panjf2000/ants/[email protected]/worker.go:67\n  | runtime.goexit\n  | \t/usr/local/go/src/runtime/asm_amd64.s:1598\nWraps: (2) failed to query\nWraps: (3) segment lacks[segment=451522410132294501]: channel not available[channel=by-dev-rootcoord-dml_7_451522410132294244v0]\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"]
[2024/09/09 07:29:22.190 +00:00] [WARN] [proxy/impl.go:3505] ["Query failed to WaitToFinish"] [traceID=355d2f946b7c58cbb1c075eacc54730f] [role=proxy] [db=default] [collection=newCollection] [partitions="[]"] [useDefaultConsistency=false] [error="failed to query: segment lacks[segment=451522410132294501]: channel not available[channel=by-dev-rootcoord-dml_7_451522410132294244v0]"] [errorVerbose="failed to query: segment lacks[segment=451522410132294501]: channel not available[channel=by-dev-rootcoord-dml_7_451522410132294244v0]\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/internal/proxy.(*queryTask).Execute\n  | \t/go/src/github.com/milvus-io/milvus/internal/proxy/task_query.go:463\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).processTask\n  | \t/go/src/github.com/milvus-io/milvus/internal/proxy/task_scheduler.go:466\n  | github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).queryLoop.func1\n  | \t/go/src/github.com/milvus-io/milvus/internal/proxy/task_scheduler.go:545\n  | github.com/milvus-io/milvus/pkg/util/conc.(*Pool[...]).Submit.func1\n  | \t/go/src/github.com/milvus-io/milvus/pkg/util/conc/pool.go:81\n  | github.com/panjf2000/ants/v2.(*goWorker).run.func1\n  | \t/go/pkg/mod/github.com/panjf2000/ants/[email protected]/worker.go:67\n  | runtime.goexit\n  | \t/usr/local/go/src/runtime/asm_amd64.s:1598\nWraps: (2) failed to query\nWraps: (3) segment lacks[segment=451522410132294501]: channel not available[channel=by-dev-rootcoord-dml_7_451522410132294244v0]\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"]

We use kafka as message broker and BOS(Baidu Object Storage) as S3 service.
Upon checking S3, the corresponding folder under the path is missing. If we rule out S3 data loss, what could potentially cause the entire segment to be deleted? Thanks for any helps.

yhmo · 2024-09-09T08:43:30Z

yhmo
Sep 9, 2024
Collaborator

So far as I know, some old versions(2.2.x, earlier versions of 2.3.x) could lead to data loss due to GC bugs. For v2.4.x, we didn't get any reports like yours.

14 replies

elpsycongroo Oct 25, 2024
Author

@yhmo Hello, today data was lost once again. Upon inspection of the S3 logs, this time our operations team discovered that it was a delete request initiated by Milvus that led to the corresponding segment loss. At this point, we can likely rule out any issues with S3. Therefore, I need to reopen this discussion: why would Milvus perform such an action?

xiaofan-luan Oct 26, 2024
Maintainer

Milvus dropped the data in two cases:

the collection is dropped
compaction is happened and after a period GC happened.

We need full logs for rootcoord and datacoord to figure out what happened.
But generally we didn't heard any of feedback so far that the latest 2.3 or 2.4 has a possiblity to delete data.

xiaofan-luan Oct 26, 2024
Maintainer

please open an issue with detailed log.

here is how you can get https://github.com/milvus-io/milvus/tree/master/deployments/export-log

elpsycongroo Oct 31, 2024
Author

@xiaofan-luan ETCD and Kafka are not configured in the K8s cluster; they are deployed on virtual machines. Could insufficient resources lead to GC mistakenly deleting segments? The same Milvus configuration in the production environment did not encounter this issue (ETCD and S3 clusters are different). I am planning to increase the resource allocation for the PODs and deploy ETCD and Kafka on higher-spec virtual machines to see if this can resolve the problem.
If not, we would try to export the detail log.

xiaofan-luan Oct 31, 2024
Maintainer

@xiaofan-luan ETCD and Kafka are not configured in the K8s cluster; they are deployed on virtual machines. Could insufficient resources lead to GC mistakenly deleting segments? The same Milvus configuration in the production environment did not encounter this issue (ETCD and S3 clusters are different). I am planning to increase the resource allocation for the PODs and deploy ETCD and Kafka on higher-spec virtual machines to see if this can resolve the problem. If not, we would try to export the detail log.

This is definitely not related to resources.

is there two milvus cluster trying to share same S3?
They need to config with different bucket

Reidddddd · 2024-10-29T06:18:58Z

Reidddddd
Oct 29, 2024

I can provide another view.

Do you have other clusters (e.g test-cluster, development cluster) using the same ETCD with a same rootPath？(should use a different one) It will lead to meta data mess-up and inconsistency, and GC will probably delete wrong files.

3 replies

xiaofan-luan Oct 29, 2024
Maintainer

I can provide another view.

Do you have other clusters (e.g test-cluster, development cluster) using the same ETCD with a same rootPath？(should use a different one) It will lead to meta data mess-up and inconsistency, and GC will probably delete wrong files.

ye s

xiaofan-luan Oct 29, 2024
Maintainer

I can provide another view.
Do you have other clusters (e.g test-cluster, development cluster) using the same ETCD with a same rootPath？(should use a different one) It will lead to meta data mess-up and inconsistency, and GC will probably delete wrong files.

yes, that is also some mistake we saw very common

elpsycongroo Oct 31, 2024
Author

Thank you for providing that opinion, but the ETCD is a new cluster and only milvus's running on it :-(

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

What can cause data loss in S3? #36112

{{title}}

Replies: 2 comments 17 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

Select a reply

What can cause data loss in S3? #36112

elpsycongroo Sep 9, 2024

Replies: 2 comments · 17 replies

yhmo Sep 9, 2024 Collaborator

elpsycongroo Oct 25, 2024 Author

xiaofan-luan Oct 26, 2024 Maintainer

xiaofan-luan Oct 26, 2024 Maintainer

elpsycongroo Oct 31, 2024 Author

xiaofan-luan Oct 31, 2024 Maintainer

Reidddddd Oct 29, 2024

xiaofan-luan Oct 29, 2024 Maintainer

xiaofan-luan Oct 29, 2024 Maintainer

elpsycongroo Oct 31, 2024 Author

elpsycongroo
Sep 9, 2024

Replies: 2 comments 17 replies

yhmo
Sep 9, 2024
Collaborator

elpsycongroo Oct 25, 2024
Author

xiaofan-luan Oct 26, 2024
Maintainer

xiaofan-luan Oct 26, 2024
Maintainer

elpsycongroo Oct 31, 2024
Author

xiaofan-luan Oct 31, 2024
Maintainer

Reidddddd
Oct 29, 2024

xiaofan-luan Oct 29, 2024
Maintainer

xiaofan-luan Oct 29, 2024
Maintainer

elpsycongroo Oct 31, 2024
Author