[WIP] Expose extend() in C API

[ajit283]

Add functionality to add additional vectors after build to C API

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[benfred]

/ok to test

[benfred]

thanks for adding this functionality! This change looks good to me -

[benfred]

This doesn't need to be changed in this PR (especially since the rest of cagra/ivf-* doesn't follow this right now) - but I think we should start re-using the C-api parameter structs in the C++ api , like we are doing in the distances here

cuvs/cpp/include/cuvs/distance/distance.hpp

Line 27 in 72154b0

using DistanceType = cuvsDistanceType;

. This means we don't have to have two different identical structs in the C and C++ api, and copy between them you are doing here

[benfred]

for debugging the segfault - does this still occur when called with the original dataset - like :

Suggested change

	cuvsCagraExtend(res, extend_params, &additional_dataset_tensor, index);
	cuvsCagraExtend(res, extend_params, &dataset_tensor, index);

[ajit283]

still segfaults unfortunately

[cjnolet]

@ajit283 I think @benfred might be right about this being from the number of vectors in the dataset. Can we generate a matrix using the make_blobs from raft::random ? Then we can just generate a larger dataset and bypass the issue.

[cjnolet]

/ok to test

[cjnolet]

@ajit283 I think we discussed generating a bigger dataset for the test case here. You could even use raft::random::make_blobs to accomplish this.

[ajit283]

👍 will look into this tomorrow

[cjnolet]

/ok to test

[cjnolet]

@ajit283 can you fix the conflicts in this PR?

[ajit283]

Done! We should look into the tests for this one.

• The previous approach was to test against a set of precomputed queries, expected neighbors and distances.
• This does not work anymore as we generate the dataset dynamically using make_blobs(). Right now I am adding a static set of vectors to the dataset using extend(), then query these vectors and check if I get them back with distance 0. This shows that extend() works, but is generally a regression compared to the previous tests.
• My ideas for improving the tests would be either to remove make_blobs() and use a larger static dataset, or to calculate expected neighbors and distances at runtime using brute-force.

What do you think?

[cjnolet]

/ok to test

[cjnolet]

/ok to test

[ajit283]

@cjnolet I added a test using the pairwise distance matrix as discussed in the call