[Bug]: apsi_test测试时，若接收方的数据量达到某一阈值程序会卡在某处，不执行也不终止

[WandQ]

Describe the bug

首次在本地16G的mac上，执行项目中的psi/apsi/apsi_test.cc代码无误，但当我将TestParams{100, 100000, false}修改为TestParams{10000, 100000, false}后，程序会卡住，日志如下：

[2024-07-12 07:46:20.569] [info] [ecdh_oprf_selector.cc:33] use fourq
[2024-07-12 07:46:20.697] [info] [sender_memdb.cc:696] Start inserting 5000 items in SenderMemDB
[2024-07-12 07:46:20.697] [info] [thread_pool.cc:30] Create a fixed thread pool with size 7
[2024-07-12 07:46:20.719] [info] [sender_memdb.cc:730] Found 5000 new items to insert in SenderDB
[2024-07-12 07:46:20.721] [info] [sender_memdb.cc:335] Launching 8 insert-or-assign worker tasks
[2024-07-12 07:46:20.727] [info] [sender_memdb.cc:351] Finished insert-or-assign worker tasks
[2024-07-12 07:46:20.728] [info] [sender_memdb.cc:748] Finished inserting 5000 items in SenderDB
[2024-07-12 07:46:20.729] [info] [sender_memdb.cc:696] Start inserting 5000 items in SenderMemDB
[2024-07-12 07:46:20.750] [info] [sender_memdb.cc:730] Found 5000 new items to insert in SenderDB
[2024-07-12 07:46:20.753] [info] [sender_memdb.cc:335] Launching 8 insert-or-assign worker tasks
[2024-07-12 07:46:20.758] [info] [sender_memdb.cc:351] Finished insert-or-assign worker tasks
[2024-07-12 07:46:20.758] [info] [sender_memdb.cc:748] Finished inserting 5000 items in SenderDB
[2024-07-12 07:46:20.759] [info] [sender_memdb.cc:696] Start inserting 5000 items in SenderMemDB
[2024-07-12 07:46:20.781] [info] [sender_memdb.cc:730] Found 5000 new items to insert in SenderDB
[2024-07-12 07:46:20.784] [info] [sender_memdb.cc:335] Launching 8 insert-or-assign worker tasks
[2024-07-12 07:46:20.788] [info] [sender_memdb.cc:351] Finished insert-or-assign worker tasks
[2024-07-12 07:46:20.789] [info] [sender_memdb.cc:748] Finished inserting 5000 items in SenderDB
[2024-07-12 07:46:20.790] [info] [sender_memdb.cc:696] Start inserting 5000 items in SenderMemDB
[2024-07-12 07:46:20.811] [info] [sender_memdb.cc:730] Found 5000 new items to insert in SenderDB
[2024-07-12 07:46:20.814] [info] [sender_memdb.cc:335] Launching 8 insert-or-assign worker tasks
[2024-07-12 07:46:20.818] [info] [sender_memdb.cc:351] Finished insert-or-assign worker tasks
[2024-07-12 07:46:20.819] [info] [sender_memdb.cc:748] Finished inserting 5000 items in SenderDB
[2024-07-12 07:46:20.820] [info] [sender_memdb.cc:696] Start inserting 5000 items in SenderMemDB
[2024-07-12 07:46:20.842] [info] [sender_memdb.cc:730] Found 5000 new items to insert in SenderDB
[2024-07-12 07:46:20.845] [info] [sender_memdb.cc:335] Launching 8 insert-or-assign worker tasks
[2024-07-12 07:46:20.849] [info] [sender_memdb.cc:351] Finished insert-or-assign worker tasks
[2024-07-12 07:46:20.849] [info] [sender_memdb.cc:748] Finished inserting 5000 items in SenderDB
[2024-07-12 07:46:20.850] [info] [sender_memdb.cc:696] Start inserting 5000 items in SenderMemDB
[2024-07-12 07:46:20.874] [info] [sender_memdb.cc:730] Found 5000 new items to insert in SenderDB
[2024-07-12 07:46:20.877] [info] [sender_memdb.cc:335] Launching 8 insert-or-assign worker tasks
[2024-07-12 07:46:20.882] [info] [sender_memdb.cc:351] Finished insert-or-assign worker tasks
[2024-07-12 07:46:20.883] [info] [sender_memdb.cc:748] Finished inserting 5000 items in SenderDB
[2024-07-12 07:46:20.884] [info] [sender_memdb.cc:696] Start inserting 5000 items in SenderMemDB
[2024-07-12 07:46:20.907] [info] [sender_memdb.cc:730] Found 5000 new items to insert in SenderDB
[2024-07-12 07:46:20.910] [info] [sender_memdb.cc:335] Launching 8 insert-or-assign worker tasks
[2024-07-12 07:46:20.915] [info] [sender_memdb.cc:351] Finished insert-or-assign worker tasks
[2024-07-12 07:46:20.915] [info] [sender_memdb.cc:748] Finished inserting 5000 items in SenderDB
[2024-07-12 07:46:20.916] [info] [sender_memdb.cc:696] Start inserting 5000 items in SenderMemDB
[2024-07-12 07:46:20.942] [info] [sender_memdb.cc:730] Found 5000 new items to insert in SenderDB
[2024-07-12 07:46:20.945] [info] [sender_memdb.cc:335] Launching 8 insert-or-assign worker tasks
[2024-07-12 07:46:20.947] [info] [sender_memdb.cc:351] Finished insert-or-assign worker tasks
[2024-07-12 07:46:20.948] [info] [sender_memdb.cc:748] Finished inserting 5000 items in SenderDB
[2024-07-12 07:46:20.949] [info] [sender_memdb.cc:696] Start inserting 5000 items in SenderMemDB
[2024-07-12 07:46:20.971] [info] [sender_memdb.cc:730] Found 5000 new items to insert in SenderDB
[2024-07-12 07:46:20.974] [info] [sender_memdb.cc:335] Launching 8 insert-or-assign worker tasks
[2024-07-12 07:46:20.978] [info] [sender_memdb.cc:351] Finished insert-or-assign worker tasks
[2024-07-12 07:46:20.978] [info] [sender_memdb.cc:748] Finished inserting 5000 items in SenderDB
[2024-07-12 07:46:20.979] [info] [sender_memdb.cc:696] Start inserting 5000 items in SenderMemDB
[2024-07-12 07:46:21.001] [info] [sender_memdb.cc:730] Found 5000 new items to insert in SenderDB
[2024-07-12 07:46:21.004] [info] [sender_memdb.cc:335] Launching 8 insert-or-assign worker tasks
[2024-07-12 07:46:21.006] [info] [sender_memdb.cc:351] Finished insert-or-assign worker tasks
[2024-07-12 07:46:21.007] [info] [sender_memdb.cc:748] Finished inserting 5000 items in SenderDB
[2024-07-12 07:46:21.008] [info] [sender_memdb.cc:696] Start inserting 5000 items in SenderMemDB
[2024-07-12 07:46:21.029] [info] [sender_memdb.cc:730] Found 5000 new items to insert in SenderDB
[2024-07-12 07:46:21.032] [info] [sender_memdb.cc:335] Launching 8 insert-or-assign worker tasks
[2024-07-12 07:46:21.034] [info] [sender_memdb.cc:351] Finished insert-or-assign worker tasks
[2024-07-12 07:46:21.034] [info] [sender_memdb.cc:748] Finished inserting 5000 items in SenderDB
[2024-07-12 07:46:21.036] [info] [sender_memdb.cc:696] Start inserting 5000 items in SenderMemDB
[2024-07-12 07:46:21.055] [info] [sender_memdb.cc:730] Found 5000 new items to insert in SenderDB
[2024-07-12 07:46:21.058] [info] [sender_memdb.cc:335] Launching 8 insert-or-assign worker tasks
[2024-07-12 07:46:21.060] [info] [sender_memdb.cc:351] Finished insert-or-assign worker tasks
[2024-07-12 07:46:21.061] [info] [sender_memdb.cc:748] Finished inserting 5000 items in SenderDB
[2024-07-12 07:46:21.062] [info] [sender_memdb.cc:696] Start inserting 5000 items in SenderMemDB
[2024-07-12 07:46:21.117] [info] [sender_memdb.cc:730] Found 5000 new items to insert in SenderDB
[2024-07-12 07:46:21.119] [info] [sender_memdb.cc:335] Launching 8 insert-or-assign worker tasks
[2024-07-12 07:46:21.121] [info] [sender_memdb.cc:351] Finished insert-or-assign worker tasks
[2024-07-12 07:46:21.122] [info] [sender_memdb.cc:748] Finished inserting 5000 items in SenderDB
[2024-07-12 07:46:21.123] [info] [sender_memdb.cc:696] Start inserting 5000 items in SenderMemDB
[2024-07-12 07:46:21.144] [info] [sender_memdb.cc:730] Found 5000 new items to insert in SenderDB
[2024-07-12 07:46:21.147] [info] [sender_memdb.cc:335] Launching 8 insert-or-assign worker tasks
[2024-07-12 07:46:21.149] [info] [sender_memdb.cc:351] Finished insert-or-assign worker tasks
[2024-07-12 07:46:21.149] [info] [sender_memdb.cc:748] Finished inserting 5000 items in SenderDB
[2024-07-12 07:46:21.151] [info] [sender_memdb.cc:696] Start inserting 5000 items in SenderMemDB
[2024-07-12 07:46:21.172] [info] [sender_memdb.cc:730] Found 5000 new items to insert in SenderDB
[2024-07-12 07:46:21.174] [info] [sender_memdb.cc:335] Launching 8 insert-or-assign worker tasks
[2024-07-12 07:46:21.178] [info] [sender_memdb.cc:351] Finished insert-or-assign worker tasks
[2024-07-12 07:46:21.178] [info] [sender_memdb.cc:748] Finished inserting 5000 items in SenderDB
[2024-07-12 07:46:21.179] [info] [sender_memdb.cc:696] Start inserting 5000 items in SenderMemDB
[2024-07-12 07:46:21.199] [info] [sender_memdb.cc:730] Found 5000 new items to insert in SenderDB
[2024-07-12 07:46:21.202] [info] [sender_memdb.cc:335] Launching 8 insert-or-assign worker tasks
[2024-07-12 07:46:21.205] [info] [sender_memdb.cc:351] Finished insert-or-assign worker tasks
[2024-07-12 07:46:21.205] [info] [sender_memdb.cc:748] Finished inserting 5000 items in SenderDB
[2024-07-12 07:46:21.207] [info] [sender_memdb.cc:696] Start inserting 5000 items in SenderMemDB
[2024-07-12 07:46:21.230] [info] [sender_memdb.cc:730] Found 5000 new items to insert in SenderDB
[2024-07-12 07:46:21.233] [info] [sender_memdb.cc:335] Launching 8 insert-or-assign worker tasks
[2024-07-12 07:46:21.236] [info] [sender_memdb.cc:351] Finished insert-or-assign worker tasks
[2024-07-12 07:46:21.237] [info] [sender_memdb.cc:748] Finished inserting 5000 items in SenderDB
[2024-07-12 07:46:21.238] [info] [sender_memdb.cc:696] Start inserting 5000 items in SenderMemDB
[2024-07-12 07:46:21.263] [info] [sender_memdb.cc:730] Found 5000 new items to insert in SenderDB
[2024-07-12 07:46:21.266] [info] [sender_memdb.cc:335] Launching 8 insert-or-assign worker tasks
[2024-07-12 07:46:21.268] [info] [sender_memdb.cc:351] Finished insert-or-assign worker tasks
[2024-07-12 07:46:21.269] [info] [sender_memdb.cc:748] Finished inserting 5000 items in SenderDB
[2024-07-12 07:46:21.271] [info] [sender_memdb.cc:696] Start inserting 5000 items in SenderMemDB
[2024-07-12 07:46:21.296] [info] [sender_memdb.cc:730] Found 5000 new items to insert in SenderDB
[2024-07-12 07:46:21.299] [info] [sender_memdb.cc:335] Launching 8 insert-or-assign worker tasks
[2024-07-12 07:46:21.302] [info] [sender_memdb.cc:351] Finished insert-or-assign worker tasks
[2024-07-12 07:46:21.302] [info] [sender_memdb.cc:748] Finished inserting 5000 items in SenderDB
[2024-07-12 07:46:21.304] [info] [sender_memdb.cc:696] Start inserting 5000 items in SenderMemDB
[2024-07-12 07:46:21.331] [info] [sender_memdb.cc:730] Found 5000 new items to insert in SenderDB
[2024-07-12 07:46:21.334] [info] [sender_memdb.cc:335] Launching 8 insert-or-assign worker tasks
[2024-07-12 07:46:21.337] [info] [sender_memdb.cc:351] Finished insert-or-assign worker tasks
[2024-07-12 07:46:21.337] [info] [sender_memdb.cc:748] Finished inserting 5000 items in SenderDB
[2024-07-12 07:46:21.338] [info] [sender_memdb.cc:945] count: 20, item_count:100000
[2024-07-12 07:46:21.338] [info] [sender_memdb.cc:526] Start generating bin bundle caches

到上面最后一条日志就卡住了，此时本机内存情况也是正常的。

然后我更换了一台内存更大的电脑，TestParams{10000, 100000, false}能够顺利执行，但TestParams{100000, 100000, false}仍然会出现上述问题。此外，卡住之后不会大量占用内存和CPU。

Steps To Reproduce

运行命令 bazel run -c opt //psi/apsi:apsi_test

Expected behavior

期望修复该问题，接收方10000的查询数据，应该也不是很大。

Version

v0.4.0.dev240524

Operating system

macbook m1 16G

Hardware Resources

8c16g

[zimu-yuxi]

尝试使用https://github.com/secretflow/psi/tree/release/0.4.x此版本代码试下呢。
我们内部也确认下

[WandQ]

bazel run -c opt //psi/apsi:apsi_test

release/0.4.x版本一样的有这个问题

[WandQ]

尝试使用https://github.com/secretflow/psi/tree/release/0.4.x此版本代码试下呢。 我们内部也确认下

hi,关于这个问题，有结论是什么原因导致的吗？