Can milvus benefit from indexes which support update/delete

[gland1]

Hi
I would like to discuss whether milvus can benefit from indices such as fresh DISKANN which supports update and delete.
From what I see, milvus takes care of updates outside of indices scope.
Its nice that the feature is index-agnostic but in some cases this can yield low QPS (for example when many records are deleted, diskann is starting brutforce search)
What if the index has support for updates, will it be to much of an architecture change in milvus to benefit from such support ?

[yhmo]

Milvus itself has the machinery to manage update/delete. An index that supports update/delete doesn't help milvus to manage the original data.
For each segment, an index is stored in storage as an immutable file, even some entities of the segments are deleted. To avoid "write amplification", We don't intend to rewrite the index file into storage again with any update.

[liliu-z]

For insertion:
To ensure insertion speed, Milvus doesn't insert data directly to a growable index (saying fresh diskann, HNSW, IVF-series). Instead, it use BF or create a temp index (IVF, insertion speed is biggest factor to take into consideration, so graph cannot make) for these new joining data. Once the temp index is big enough, it async creates the ultimate index and replace the temp one.
For Deletion:
Milvus do soft deletion (mask deleted data as invalid) and async rebuild segment(unit data structure of vector) and its index if needed. Yes, as mentioned if too much deletion and the background compaction hasn't happened yet, the performance can be affected. But scalar filtering can also bring this trouble.