FAISSDocumentStore concurrency error

[apillai-mitre]

Hi!

I am using FAISSDocumentStore to store the documents and embeddings used by an extractive Q&A pipeline. We are trying to scale the app such that multiple users can ask the model questions and upload PDFs that the model can use concurrently. To test this, I tried uploading a PDF to the database on one browser, and shortly after starting that process I tried uploading another PDF to the database on another browser. The following is what the code for uploading a PDF to the FAISSDocumentStore looks like (it writes the new docs, updates the embeddings, and then saves):

When the second browser finishes updating its embeddings, it shows the following error on my Streamlit page for the app:

IntegrityError: (sqlite3.IntegrityError) UNIQUE constraint failed: document.index, document.vector_id [SQL: UPDATE document SET vector_id=CASE document.id WHEN ? THEN ? WHEN ? THEN ? WHEN ? THEN ? WHEN ? THEN ? WHEN ? THEN ? WHEN ? THEN ? WHEN ? THEN ? WHEN ? THEN ? WHEN ? THEN ? WHEN ? THEN ? WHEN ? THEN ? WHEN ? THEN ? WHEN ? THEN ? WHEN ? THEN ? WHEN ? THEN ? WHEN ? THEN ? WHEN ? THEN ? WHEN ? THEN ? WHEN ? THEN ? WHEN ? THEN ? WHEN ? THEN ? WHEN ? THEN ? WHEN ? THEN ? WHEN ? THEN ? WHEN ? THEN ? WHEN ? THEN ? WHEN ? THEN ? WHEN ? THEN ? WHEN ? THEN ? WHEN ? THEN ? WHEN ? THEN ? WHEN ? THEN ? WHEN ? THEN ? WHEN ? THEN ? WHEN ? THEN ? WHEN ? THEN ? WHEN ? THEN ? WHEN ? THEN ? WHEN ? THEN ? WHEN ? THEN ? WHEN ? THEN ? WHEN ? THEN ? WHEN ? THEN ? WHEN ? THEN ? WHEN ? THEN ? WHEN ? THEN ? WHEN ? THEN ? WHEN ? THEN ? WHEN ? THEN ? WHEN ? THEN ? WHEN ? THEN ? WHEN ? THEN ? WHEN ? THEN ? WHEN ? THEN ? WHEN ? THEN ? WHEN ? THEN ? WHEN ? THEN ? WHEN ? THEN ? WHEN ? THEN ? WHEN ? THEN ? WHEN ? THEN ? WHEN ? THEN ? WHEN ? THEN ? WHEN ? THEN ? WHEN ? THEN ? WHEN ? THEN ? WHEN ? THEN ? WHEN ? THEN ? WHEN ? THEN ? WHEN ? THEN ? WHEN ? THEN ? WHEN ? THEN ? WHEN ? THEN ? WHEN ? THEN ? WHEN ? THEN ? WHEN ? THEN ? WHEN ? THEN ? WHEN ? THEN ? WHEN ? THEN ? WHEN ? THEN ? WHEN ? THEN ? WHEN ? THEN ? WHEN ? THEN ? WHEN ? THEN ? WHEN ? THEN ? WHEN ? THEN ? WHEN ? THEN ? WHEN ? THEN ? WHEN ? THEN ? WHEN ? THEN ? WHEN ? THEN ? WHEN ? THEN ? WHEN ? THEN ? WHEN ? THEN ? WHEN ? THEN ? WHEN ? THEN ? WHEN ? THEN ? WHEN ? THEN ? WHEN ? THEN ? WHEN ? THEN ? WHEN ? THEN ? WHEN ? THEN ? WHEN ? THEN ? WHEN ? THEN ? WHEN ? THEN ? WHEN ? THEN ? WHEN ? THEN ? WHEN ? THEN ? END WHERE document.id IN (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) AND document."index" = ?] [parameters: ('127749aed5e5941b9395e6d5724db4ee', '0', '12f26bbb83cf68c779ace0f3a9a32544', '1', '1735815b47ce9bb6453bc1f5461d1bbf', '2', '19f9c7d04784d1fd5a872f7551fc6309', '3', '1bb1fa307ae29317096ac0e0969b5b06', '4', '1c5f4e778f52b8dbc09a1570f6999a9d', '5', '1ed7a9b24578fd0696e615c3704ebf45', '6', '20a693ac31e9053501cd7e87efed76d3', '7', '22f13cd3db03154d06df2d33a25aea25', '8', '274bb52fbc20e15f4bd4a7a62ea7e62d', '9', '27d2411b3bd6ec736d59a452a7655a1', '10', '29f8f6bc7c7f19f16d97392fc8624c92', '11', '2db19adc1bd57b59a74b8ea631e43ca8', '12', '2f44af079c9400d20ddc5c23b3701c02', '13', '32db5bf13e5e8241db630874ec4573a4', '14', '33f2e8d8e6a55874fce0902c520fab5a', '15', '3761342a1c4d41757c5c51ba411c9b04', '16', '37629be4e095f0308c1a511d26834f8d', '17', '38e29bf3ef51e75f9c7e6299ac94aa9b', '18', '3956cb33aac3db9697b30b678841a62a', '19', '3a109cc2c61e598e7349f01969c03e48', '20', '3a2ec6d844b134f3f2c27b59729706f0', '21', '3ad46697dfce9c5970f75272e9d1c1c', '22', '3b743fe303df6d58ea2e3d70dc97dbb3', '23', '3b7d9066e60a93f4b58d0e532e8ab60a', '24', '3cba6233bb9b414a0bb9e2d8a51a23c7', '25', '3df243dd0a4d59a72241a58a31e240cb', '26', '422957c6ec8782915cc69ed2e3dee549', '27', '42b1fb27dd0a07b807773060eae11b79', '28', '4337821ccb54a60ef64bd117a4903eac', '29', '43ca4caad8ce9e4b897be6fd85c3bf72', '30', '464aa8c0c6a639031607ed2062ddaa4c', '31', '49d536766ae567228dbc19e2980ef7be', '32', '4dacd0644a12c762259909269f5f7be1', '33', '5092c569252a6db864b0dcd7349d8b7f', '34', '545d7129e4594759fe65366943b4e690', '35', '54d3822d7f1f41e3ef0af75b78f4ba0e', '36', '56151c451384b09b398a6487de56e5c2', '37', '5bae5155ca6b40669d35d1b8520601c2', '38', '5dcb992917ca74b065247ece49feb3fb', '39', '60146c38c4549c21a44dc4e1bb52bf42', '40', '620e187c8105ea29af31b49190b27eba', '41', '631985caaf8090873394f66755fce3a7', '42', '6522fb45e86951b3ab482104c8186b48', '43', '670f245357d7715d1f69f2883b17c93c', '44', '6a1430f1a569cfd249a4daac61441f6c', '45', '6b95dd96e31a46e8ade35f2c6112e99', '46', '6d5c078b65f2b8288e2fc7ce6682f976', '47', '7014c9034aa6857cb37c7b5c51d24de', '48', '722e91d09b90104b5c987de34adbc06a', '49', '72fa5c803fe93a4af9f19b92a5e188c9', '50', '788fee5edb1d1845392ed89a4edbefa', '51', '793ec4424a1c81733158762794a73852', '52', '7c3a0f24133ade1b00affefda6ea061c', '53', '7d6cb55cd159726363f07f2accdecfce', '54', '7e4efa3c402a84cf8cd5d37e20182521', '55', '7f3be938271db57141ac12837ed37805', '56', '80228a6e3b0bbaa24a923752f7fe8da2', '57', '806c1cc9b535eebbb1bcfda21f83502e', '58', '811943c5c563cf290e7f1fb749eb81ba', '59', '847b6bf2787603807fd1da132c5bce9e', '60', '84b810cded6c4a8c9403a069795aac90', '61', '86e263870dfdc4efcb989aaf36b1f3fc', '62', '8894274c86812c7d1c8872a6d6fb9b00', '63', '896f831ac61629353868ea73edc28114', '64', '8ba2e4bd853ea4e4d00f9c466b277f50', '65', '9037303b1b8b2decb2006b2ac792e434', '66', '9321516a5b65bf8e5f97807ce229d1b8', '67', '97a196e1e10b7387551c940e6c3eb9e9', '68', '990fb12cde9715808b1c380072b41432', '69', '9cde1dc1543b5470029d5e4e1f78eafc', '70', '9e902b91f208ac542c2857ad1bc77a1c', '71', 'a2747eb4731d3985a894c25318a6cdc6', '72', 'a3a0db21b0779742ce78d3d777edf39f', '73', 'a52217de36bb872b80d7f0fc7d8f689f', '74', 'a54352700d5b29984d2b3f9ba81d9e46', '75', 'aaf49dfb62018e9a8ff8b59092b152e6', '76', 'ad862a91ac8a960b5026ea0d7471882f', '77', 'ae80b533cda9ca921489828c1a143fc5', '78', 'b08d4056c47e7347c184aa4d44404387', '79', 'b34c068187d22b5439a59e35a0ade496', '80', 'b6b320e83421d986af032176e6cb26f5', '81', 'be11a1442f7c3a7749b7ae1361fc35fe', '82', 'c0fcb52d5e83d6dbe34e615fbae259d8', '83', 'c1a48bd2e4e547fd3d296b0145107336', '84', 'c354e3fc42ca30f03d91b06e58f49228', '85', 'c37aecfe5f3ae782b41495b26008e263', '86', 'c9826febff77c9108c43e4d8cbc88be2', '87', 'cdb3ba8954dd23566e7b53320ca0dda3', '88', 'ce7fb9407cddfaca39ffc5f026422058', '89', 'd016cc7ad0c45a43e2bc797987b9e5f', '90', 'd287cd2ef667bc82e678aed598fb67b9', '91', 'd2fd30a95d14838fef190bd4380e84f3', '92', 'd6d566e74a6e7c3a09f96425a8bd6b54', '93', 'd8bf9c4fd9b29a004696429303bb3349', '94', 'dc1f78bb5fec08a9f384da0e04608ec4', '95', 'dcdce163c2c4aac612b5acc9f2a3ce78', '96', 'de60fd525de7b2f4c63c008bdaf00cfa', '97', 'e0195b60cba8e9933614abf38a7e98eb', '98', 'e0598789699549958ac6ba85d2798f9e', '99', 'e469a33d5e2e949a6d80888c2ffb777d', '100', 'e8b09ab504d3f2eff49bc36f669f4f2c', '101', 'ec92ba21d958ab3e6f4fcc7464990e1e', '102', 'ee275794a71c71dff434f49c4bf9f793', '103', 'eea92b5c552889e86daa4c88596d3c56', '104', 'f83d5aef25502fba227a55191457cbd1', '105', 'fa3edd220f24e880b578f4a0d29df61a', '106', 'fb66ad03f8858dc32f41ed0b213bbe7f', '107', '127749aed5e5941b9395e6d5724db4ee', '12f26bbb83cf68c779ace0f3a9a32544', '1735815b47ce9bb6453bc1f5461d1bbf', '19f9c7d04784d1fd5a872f7551fc6309', '1bb1fa307ae29317096ac0e0969b5b06', '1c5f4e778f52b8dbc09a1570f6999a9d', '1ed7a9b24578fd0696e615c3704ebf45', '20a693ac31e9053501cd7e87efed76d3', '22f13cd3db03154d06df2d33a25aea25', '274bb52fbc20e15f4bd4a7a62ea7e62d', '27d2411b3bd6ec736d59a452a7655a1', '29f8f6bc7c7f19f16d97392fc8624c92', '2db19adc1bd57b59a74b8ea631e43ca8', '2f44af079c9400d20ddc5c23b3701c02', '32db5bf13e5e8241db630874ec4573a4', '33f2e8d8e6a55874fce0902c520fab5a', '3761342a1c4d41757c5c51ba411c9b04', '37629be4e095f0308c1a511d26834f8d', '38e29bf3ef51e75f9c7e6299ac94aa9b', '3956cb33aac3db9697b30b678841a62a', '3a109cc2c61e598e7349f01969c03e48', '3a2ec6d844b134f3f2c27b59729706f0', '3ad46697dfce9c5970f75272e9d1c1c', '3b743fe303df6d58ea2e3d70dc97dbb3', '3b7d9066e60a93f4b58d0e532e8ab60a', '3cba6233bb9b414a0bb9e2d8a51a23c7', '3df243dd0a4d59a72241a58a31e240cb', '422957c6ec8782915cc69ed2e3dee549', '42b1fb27dd0a07b807773060eae11b79', '4337821ccb54a60ef64bd117a4903eac', '43ca4caad8ce9e4b897be6fd85c3bf72', '464aa8c0c6a639031607ed2062ddaa4c', '49d536766ae567228dbc19e2980ef7be', '4dacd0644a12c762259909269f5f7be1', '5092c569252a6db864b0dcd7349d8b7f', '545d7129e4594759fe65366943b4e690', '54d3822d7f1f41e3ef0af75b78f4ba0e', '56151c451384b09b398a6487de56e5c2', '5bae5155ca6b40669d35d1b8520601c2', '5dcb992917ca74b065247ece49feb3fb', '60146c38c4549c21a44dc4e1bb52bf42', '620e187c8105ea29af31b49190b27eba', '631985caaf8090873394f66755fce3a7', '6522fb45e86951b3ab482104c8186b48', '670f245357d7715d1f69f2883b17c93c', '6a1430f1a569cfd249a4daac61441f6c', '6b95dd96e31a46e8ade35f2c6112e99', '6d5c078b65f2b8288e2fc7ce6682f976', '7014c9034aa6857cb37c7b5c51d24de', '722e91d09b90104b5c987de34adbc06a', '72fa5c803fe93a4af9f19b92a5e188c9', '788fee5edb1d1845392ed89a4edbefa', '793ec4424a1c81733158762794a73852', '7c3a0f24133ade1b00affefda6ea061c', '7d6cb55cd159726363f07f2accdecfce', '7e4efa3c402a84cf8cd5d37e20182521', '7f3be938271db57141ac12837ed37805', '80228a6e3b0bbaa24a923752f7fe8da2', '806c1cc9b535eebbb1bcfda21f83502e', '811943c5c563cf290e7f1fb749eb81ba', '847b6bf2787603807fd1da132c5bce9e', '84b810cded6c4a8c9403a069795aac90', '86e263870dfdc4efcb989aaf36b1f3fc', '8894274c86812c7d1c8872a6d6fb9b00', '896f831ac61629353868ea73edc28114', '8ba2e4bd853ea4e4d00f9c466b277f50', '9037303b1b8b2decb2006b2ac792e434', '9321516a5b65bf8e5f97807ce229d1b8', '97a196e1e10b7387551c940e6c3eb9e9', '990fb12cde9715808b1c380072b41432', '9cde1dc1543b5470029d5e4e1f78eafc', '9e902b91f208ac542c2857ad1bc77a1c', 'a2747eb4731d3985a894c25318a6cdc6', 'a3a0db21b0779742ce78d3d777edf39f', 'a52217de36bb872b80d7f0fc7d8f689f', 'a54352700d5b29984d2b3f9ba81d9e46', 'aaf49dfb62018e9a8ff8b59092b152e6', 'ad862a91ac8a960b5026ea0d7471882f', 'ae80b533cda9ca921489828c1a143fc5', 'b08d4056c47e7347c184aa4d44404387', 'b34c068187d22b5439a59e35a0ade496', 'b6b320e83421d986af032176e6cb26f5', 'be11a1442f7c3a7749b7ae1361fc35fe', 'c0fcb52d5e83d6dbe34e615fbae259d8', 'c1a48bd2e4e547fd3d296b0145107336', 'c354e3fc42ca30f03d91b06e58f49228', 'c37aecfe5f3ae782b41495b26008e263', 'c9826febff77c9108c43e4d8cbc88be2', 'cdb3ba8954dd23566e7b53320ca0dda3', 'ce7fb9407cddfaca39ffc5f026422058', 'd016cc7ad0c45a43e2bc797987b9e5f', 'd287cd2ef667bc82e678aed598fb67b9', 'd2fd30a95d14838fef190bd4380e84f3', 'd6d566e74a6e7c3a09f96425a8bd6b54', 'd8bf9c4fd9b29a004696429303bb3349', 'dc1f78bb5fec08a9f384da0e04608ec4', 'dcdce163c2c4aac612b5acc9f2a3ce78', 'de60fd525de7b2f4c63c008bdaf00cfa', 'e0195b60cba8e9933614abf38a7e98eb', 'e0598789699549958ac6ba85d2798f9e', 'e469a33d5e2e949a6d80888c2ffb777d', 'e8b09ab504d3f2eff49bc36f669f4f2c', 'ec92ba21d958ab3e6f4fcc7464990e1e', 'ee275794a71c71dff434f49c4bf9f793', 'eea92b5c552889e86daa4c88596d3c56', 'f83d5aef25502fba227a55191457cbd1', 'fa3edd220f24e880b578f4a0d29df61a', 'fb66ad03f8858dc32f41ed0b213bbe7f', 'document')] (Background on this error at: http://sqlalche.me/e/14/gkpj)

It gives the error in reference to updating the embeddings. On restarting the app, I get the following error that prevents me from using it:

ValueError: The number of documents in the SQL database (108) doesn't match the number of embeddings in FAISS (74). Make sure your FAISS configuration file points to the same database that you used when you saved the original index.

I have tried different configurations of the code, such as writing the documents, saving them, and then updating the embeddings, but none of them seem to work. Is this a bug with FAISS, or should I switch to another document store usable with Haystack such as Milvus or Weaviate?

[ZanSara]

Hey @apillai-mitre! FAISSDocumentStore is not really designed to work at scale in its current form. I'd strongly recommend to move to another docstore, and the two you mention are great choices 🙂

[apillai-mitre]

Hi @ZanSara, thanks for your response! My preliminary tests with Weaviate have been working well so far so I think I'll switch to that.