clustering compaction partition stats

[gland1]

hi
I've examine latest code for clustering compaction.
I see that datanode is saving clustering compaction partition stats for temp segments(unsorted segments before tasks stats),
but the query node during pruning- compares segment ids from the temp list and not from the final segments which were done by stats tasks - hence no pruning is done.
Please advise
thanks
-gili

[wayblink]

@gland1 Nice catch! Clustering compaction + stats is under development. We will fix it in the public release version

[gland1]

Thanks
Any estimation as to when is it expected to be released?

[wayblink]

@xiaocai2333 is fixing it, We will let you know when it fixed in master branch

[xiaofan-luan]

hi I've examine latest code for clustering compaction. I see that datanode is saving clustering compaction partition stats for temp segments(unsorted segments before tasks stats), but the query node during pruning- compares segment ids from the temp list and not from the final segments which were done by stats tasks - hence no pruning is done. Please advise thanks -gili

is there a use case for that you want to use cluster compaction?

[gland1]

We are looking for a Milvus solution that will be capable to support search with 1B collection
And we were considering such option for improved search performance for such big collections

[xiaofan-luan]

Clustering Compaction is designed for improving searching performance.

With clustering compaction, milvus try to assign vectors based on their similarity(Similar to IVF index) and when search happened we can only search on closest topN segments. We will lose some potential candidates but you will see great performance improvement under large dataset.

This feature is still under heavy deployment and might ready to use with immutable or rare updated collections

[xiaofan-luan]

@xiaocai2333
this is a issue because partiton stats record the old segment. What will be the right fix for that?

[xiaocai2333]

pr #36408 can fix it, it will regenerate the partition stats for new segments.

[gland1]

I've tested the latest master with this fix in the following scenario:
k8s cluster of 3 nodes.
Added KMEANS implementation to knowhere.
Tested clustering compaction by vector on a 10M wiki db with DISKANN index.
With stats task enabled - I get poor recall. (&50-70%)
Without stats I get good recall.(95-98%)
In both cases I see segment pruning is done.

And BTW, there's a bug in compaction_task_clustering.go - in shadow clone, TmpSegments should be cloned as well, otherwise you reach processStats with empty list.

[xiaofan-luan]

I've tested the latest master with this fix in the following scenario: k8s cluster of 3 nodes. Added KMEANS implementation to knowhere. Tested clustering compaction by vector on a 10M wiki db with DISKANN index. With stats task enabled - I get poor recall. (&50-70%) Without stats I get good recall.(95-98%) In both cases I see segment pruning is done.

And BTW, there's a bug in compaction_task_clustering.go - in shadow clone, TmpSegments should be cloned as well, otherwise you reach processStats with empty list.

Thanks for the feedback.

You will need to create more segment. Wikipedia is a hard dataset and you will need to split 10M data into at least 64 segment and search on at least 8 of them to get a reasonable recall.

On the other side, you can increase segment size to 4G if you have enough index build node and the performance should be better.

[wayblink]

I've tested the latest master with this fix in the following scenario: k8s cluster of 3 nodes. Added KMEANS implementation to knowhere. Tested clustering compaction by vector on a 10M wiki db with DISKANN index. With stats task enabled - I get poor recall. (&50-70%) Without stats I get good recall.(95-98%) In both cases I see segment pruning is done.

And BTW, there's a bug in compaction_task_clustering.go - in shadow clone, TmpSegments should be cloned as well, otherwise you reach processStats with empty list.

Thanks for your feedback. It is a bug, I will fix it.

[gland1]

I would like to inform you about another critical bug in clustering_compactor.go(datanode)
in mappingSegment method - the offset variable is wrongly initialized within the loop while
it should be initialized outside the loop. This causes scrambling of analysis result and therefor low recall on pruned queries

[xiaofan-luan]

Hi @gland1 ,
I thought you are running on a interesting case. And thanks for the insightful suggestion.
Would you like to conversation about ur use case and we can discuss some details.
my email is [email protected]