Streaming setting

[markoarnauto]

This library offers the major building blocks to build streaming systems where single entries or mini batches are added to the index iteratively (in a reasonable amount of time). But in a streaming system the index can grow infinetly and one needs to implement some pruning mechanisms. One can delete the oldest entries and then compact the segments at a given interval but these operations seem costly. I'm wondering what the runtime performance of these operations is (might depend on the index)? If there is a certain index type that is well suited for a streaming setting? Or if it is possible to implement a capped (fifo) index structure that is automatically pruned on insertion?

[shiyu22]

As you say, deleting the oldest entries and then compact the segments is costly. Maybe you can use partition, for example, to set the partition's tag by date and then query the latest partitions, but of course, you can also delete old partitions.

[markoarnauto]

Ok thank you very much for your feedback! It's quite helpful. I have one question left regarding streaming insertion. The docs state:

In scenario with continuous data insertion, because Milvus does not index segments with a size less than index_file_size, it uses brute-force search as the query method.

Thus segments are accumulated until index_file_size is reached. If index_file_size is reached a merge operation is triggered and an index is created. But the docs also state that merge operations are triggered after each flush. So when my index_file_size is 4096 and I insert and flush 4 batches of 1024 consequtively, how many merge operations are triggered? one or four?

[shiyu22]

Hi,
The merge operation is used to merge inserted data, in other words, Milvus merges small data segments into new data segments, deletes the small data segments, and updates the metadata to reduce the disk footprint.

While indexing is independent of merge/flush operations, Milvus can automatically or manually create index:

1. When you create_collction() then create_index() and index_file_size=4096, so when you insert the data reaches 4096M Milvus will automatically create an index for the segment.
2. When you insert data then you manually call create_index(), then Milvus will create an index for the data even though the segment does not reach the index_file_size.

The merge strategy is like the 2048 game, so when your index_file_size is 4096 and insert and flush 4 batches of 1024 consecutively, Milvus merge the first two 1024s, then the last two, and finally all the data, so it's three merges. So the Segment size will be 4096 and it will be automatically indexed.

[markoarnauto]

If someone has similar issues, this article was pretty helpful.