Update vector collection tuning tips and describe efSearch hint [AI-195] [AI-192] #1521
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
✅ Deploy Preview for hardcore-allen-f5257d ready!
To edit notification comments on pull requests, go to your Netlify site configuration. |
k-jamroz
changed the title
yuce
requested changes
Feb 4, 2025
| 1. For searches with small `topK` (for example, 10) it may be beneficial to artificially increase `topK`, adjust `partitionLimit` accordingly, and discard extra results. If you need 10 results, a good starting point for tuning could be `topK=100` and a `partitionLimit` between 50 and 100. While this will make the search slower, it will also improve quality, sometimes significantly. Overall, this setup can be more efficient than increasing index build parameters (`max-degree`, `ef-construction`) which results in slower index builds and searches. With a very small `topK` or `paritionLimit`, the search algorithm is less able to escape local minima and find the best results. | ||
| 2. Vector deduplication does not incur significant overhead for uploads (usually less than 1%) and searches. You may consider disabling it to get slightly better performance and smaller memory usage if your dataset does not contain duplicated vectors. However, be aware that in the presence of many duplicated vectors with deduplication disabled, a similarity search may return poor quality results. | ||
| 3. For a given query, each vector index partition is searched by 1 thread. The number of concurrent partition searches is configured by specifying a pool size for `hz:query` executor, which by default has 16 threads per member. If optimizing for search, we recommend setting the `hz:query` pool size to be that of the physical core count of your host machines: this will result in a good balance between search throughput and CPU utilization. Setting `hz:query` to have a pool size greater than that of the physical core count will not deliver a significant increase in throughput but it will increase total CPU utilization. The `hz:query` pool size can be changed as follows: | ||
| 1. Enable Vector API. |
There was a problem hiding this comment.
It would be good to mention how to do that, by using --add-modules jdk.incubator.vector or link to the appropriate document.
| @@ -93,12 +93,24 @@ To allow the system to return enough results, the following conditions must be s | |||
|
|
|||
| - `partitionLimit * partitionCount >= topK`, `partitionLimit <= topK` | |||
| - `memberLimit * memberCount >= topK`, `memberLimit <= topK` | |||
| - `efSearch >= partitionLimit`, if `partitionLimit` is not configured explicitly this applies to the default `partitionLimit` value | |||
There was a problem hiding this comment.
AFAIS the default value of partitionLimit is not mentioned anywhere.
There was a problem hiding this comment.
the exact formula is complicated (https://github.com/hazelcast/hazelcast-mono/pull/3258). I described it generally:
partitionLimitis calculated based ontopKand cluster configuration (number of partitions)
| 4. Before increasing index build parameters (`max-degree`, `ef-construction`) which would result in slower index builds and searches and a larger index, | ||
| test if adjusting `efSearch` gives satisfactory results. | ||
| 5. Vector deduplication does not incur significant overhead for uploads (usually less than 1%) and searches. You may consider disabling it to get slightly better performance and smaller memory usage if your dataset does not contain duplicated vectors. However, be aware that in the presence of many duplicated vectors with deduplication disabled, a similarity search may return poor quality results. | ||
| 6. For a given query, each vector index partition is searched by 1 thread. The number of concurrent partition searches is configured by specifying a pool size for `hz:query` executor, which by default has 16 threads per member. If optimizing for search, we recommend setting the `hz:query` pool size to be that of the physical core count of your host machines: this will result in a good balance between search throughput and CPU utilization. Setting `hz:query` to have a pool size greater than that of the physical core count will not deliver a significant increase in throughput but it will increase total CPU utilization. The `hz:query` pool size can be changed as follows: |
There was a problem hiding this comment.
Suggested change
| 6. For a given query, each vector index partition is searched by 1 thread. The number of concurrent partition searches is configured by specifying a pool size for `hz:query` executor, which by default has 16 threads per member. If optimizing for search, we recommend setting the `hz:query` pool size to be that of the physical core count of your host machines: this will result in a good balance between search throughput and CPU utilization. Setting `hz:query` to have a pool size greater than that of the physical core count will not deliver a significant increase in throughput but it will increase total CPU utilization. The `hz:query` pool size can be changed as follows: | |
| 6. For a given query, each vector index partition is searched by one thread. The number of concurrent partition searches is configured by specifying a pool size for `hz:query` executor, which by default has 16 threads per member. If optimizing for search, we recommend setting the `hz:query` pool size to be that of the physical core count of your host machines: this will result in a good balance between search throughput and CPU utilization. Setting `hz:query` to have a pool size greater than that of the physical core count will not deliver a significant increase in throughput but it will increase total CPU utilization. The `hz:query` pool size can be changed as follows: |
There was a problem hiding this comment.
How is hz:query is changed? It would be good to link to the relevant doc.
There was a problem hiding this comment.
there is an example below
yuce
approved these changes
Feb 4, 2025
oliverhowell
requested changes
Feb 5, 2025
There was a problem hiding this comment.
Mostly minor edits but a few questions on the table values. Not suggested any changes to these pages that are not in this PR but I do see a few issues which we can come back to in a later update e.g. page name vs. title, explicitly introducing code samples (helps with Ask AI)
| @@ -718,6 +719,9 @@ You can use hints to fine-tune search precision, especially with smaller `limit` | |||
| |=== | |||
| |Hint|Description | |||
|
|
|||
| |efSearch | |||
| |Size of list of potential candidates during search. Larger value results in better precision but slower execution. | |||
There was a problem hiding this comment.
What is the format of this value? i.e. how should the user use this hint?
Can we add a type or default to the table overall?
Also we should explicitly introduce the examples in general e.g. The following code example shows how to add search options and hints
There was a problem hiding this comment.
If you add these hints to the example (not sure if you should combine them though) then this would suffice rather than adding detail about type of value needed.
There was a problem hiding this comment.
Co-authored-by: Oliver Howell <[email protected]>
…tor-tuning-6.0-part
oliverhowell
approved these changes
Feb 5, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There were important changes in 6.0 impacting vector collection tuning: introduction of
efSearchhint and default value forpartitionLimit.After https://hazelcast.atlassian.net/browse/AI-192 mutating operations do not fail during optimization.