knowhere IVF_PQ build time degrade

[cydrain]

User reports that IVF_PQ index build time degrade 60x between Milvus 2.3.20 and Milvus 2.4.6
milvus-io/milvus#34999

Test knowhere separately, can reproduce this degradation:

Commit Time	Commit ID	IVF_PQ build time	other
2023/11/09	6b2fe5	31s
2023/11/09	f7fd04	257s	PR #181, upgrade to faiss-1.7.4
2023/12/29	e1e012	239s
2024/01/12	e571ed	245s
2024/01/22	170e88	248s
2024/01/24	0f63f2	244s
2024/01/26	e9574e	394s	PR #359
2024/02/29	90553d	390s
2024/04/07	v2.3.0	406s

[cydrain]

Hi @alexanderguzhva @zhengbuqian

Please help to check this issue

[cydrain]

/assign @alexanderguzhva @zhengbuqian

[cydrain]

Can use knowhere benchmark to reproduce:

1. conan install .. --build=missing -o with_ut=True -o with_benchmark=True -s compiler.libcxx=libstdc++11 -s build_type=Release
2. conan build ..
3. cd Release/benchmark && rm -f Makefile
4. ln -s ../../../benchmark/hdf5/ref_logs/Makefile .
5. (link sift-128-euclidean.hdf5)
6. make test_float_ivf_pq
7. from the log, can see the build time is 244s

[0.202 s] Reading index file: sift-128-euclidean_IVF_PQ_1024_8.index
[0.202 s] Building all on 1000000 vectors
[244.352 s] Writing index file: sift-128-euclidean_IVF_PQ_1024_8.index

[alexanderguzhva]

The issue presented itself when we decided to wrap training and adding with a folly threadpool.

running with OPENBLAS_NUM_THREADS=1 solves the problem. The issues is caused by infamous N^2 threads use case, where each of N OpenMP threads runs sgemm(), which runs its own N threads, causing N^2 threads in total. By default, OpenBLAS uses pthread for threading, not OpenMP (check https://github.com/OpenMathLib/OpenBLAS/wiki/Faq#multi-threaded).

The correct solution is not to install libopenblas, but to install libopenblas0-openmp instead (or libopenblas-openmp-dev) for Ubuntu. Please make sure that no libopenblas0-pthread / libopenblas-pthread-dev are installed, because it is a default version. Linux users may wish to use LD_PRELOAD that points to correct openmp-version of libopenblas.

I'm not sure what happens if MKL is used as a BLAS (MKL supports OpenMP without any changes? MKL_NUM_THREADS=1 to use?)

I'm not sure what happens in BLAS is provided by Apple

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. Rotten issues close after 30d of inactivity. Reopen the issue with /reopen.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. Rotten issues close after 30d of inactivity. Reopen the issue with /reopen.