Search keeps on indexing

[conradhagemans]

I cannot use the full-text search functions. I have about 55000 files on the server. Each time you enter a search text it starts to indexing. This takes for ever.
If I switch full-text search off (admin) the results are direct shown. Ofcourse only on filename search.

[cmigliorini]

Same. Running OC 7.0.2, 50k files. Resembles issue#2 affecting OC6, which is closed (I did not see why it was closed)

[ferdiga]

Same here OC 7.0.4.2 2M files

Steps to reproduce

1. add local storage with 2m files
   2.
   3.

Expected behaviour

show search results

Actual behaviour

starts indexing and hangs

Server configuration

Operating system:
Linux
Web server:
apache2
Database:
mariadb
PHP version:
PHP 5.4.4-14+deb7u11
ownCloud version: (see ownCloud admin page)
7.0.4.2
Updated from an older ownCloud or fresh install:
updated
List of activated apps:

The content of config/config.php:

Insert your config.php content here
(Without the database password, passwordsalt and secret)

Are you using external storage, if yes which one: local/smb/sftp/...
local
Are you using encryption: yes/no
not sure what sho
uld be encrypted ?

Client configuration

Browser:
Chrome
Operating system:
Linux Ubuntu

Logs

Web server error log

Insert your webserver log here

ownCloud log (data/owncloud.log)

{"reqId":"54b24ce185af7","app":"search_lucene","message":"Invalid archive or corrupted .xlsx file. Trace:\\n#0 \/var\/www\/owncloud\/apps\/search_lucene\/3rdparty\/Zend\/Search\/Lucene\/Document\/Xlsx.php(263): Zend_Search_Lucene_Document_Xlsx->__construct('\/home\/strakosch...', false)\n#1 \/var\/www\/owncloud\/apps\/search_lucene\/lib\/indexer.php(106): Zend_Search_Lucene_Document_Xlsx::loadXlsxFile('\/home\/strakosch...')\n#2 \/var\/www\/owncloud\/apps\/search_lucene\/ajax\/lucene.php(44): OCA\\Search_Lucene\\Indexer::indexFile('\/strakosch\/dos\/...', 'gass')\n#3 \/var\/www\/owncloud\/apps\/search_lucene\/ajax\/lucene.php(76): index()\n#4 \/var\/www\/owncloud\/lib\/base.php(798): require_once('\/var\/www\/ownclo...')\n#5 [internal function]: OC::loadAppScriptFile(Array)\n#6 \/var\/www\/owncloud\/lib\/private\/route\/router.php(227): call_user_func(Array, Array)\n#7 \/var\/www\/owncloud\/lib\/base.php(706): OC\\Route\\Router->match('\/apps\/search_lu...')\n#8 \/var\/www\/owncloud\/index.php(28): OC::handleRequest()\n#9 {main}","level":3,"time":"2015-01-11T10:13:53+00:00","method":"GET","url":"\/owncloud\/index.php\/apps\/search_lucene\/ajax\/lucene.php?operation=index&requesttoken=5b86f79389f5eff37b8e"}
{"reqId":"54b24ce185af7","app":"PHP","message":"Cannot modify header information - headers already sent at \/var\/www\/owncloud\/lib\/private\/json.php#17","level":3,"time":"2015-01-11T10:13:53+00:00","method":"GET","url":"\/owncloud\/index.php\/apps\/search_lucene\/ajax\/lucene.php?operation=index&requesttoken=5b86f79389f5eff37b8e"}
{"reqId":"54b24ce185af7","app":"search_lucene","message":"Invalid archive or corrupted .xlsx file. Trace:\\n#0 \/var\/www\/owncloud\/apps\/search_lucene\/3rdparty\/Zend\/Search\/Lucene\/Document\/Xlsx.php(263): Zend_Search_Lucene_Document_Xlsx->__construct('\/home\/strakosch...', false)\n#1 \/var\/www\/owncloud\/apps\/search_lucene\/lib\/indexer.php(106): Zend_Search_Lucene_Document_Xlsx::loadXlsxFile('\/home\/strakosch...')\n#2 \/var\/www\/owncloud\/apps\/search_lucene\/ajax\/lucene.php(44): OCA\\Search_Lucene\\Indexer::indexFile('\/strakosch\/dos\/...', 'gass')\n#3 \/var\/www\/owncloud\/apps\/search_lucene\/ajax\/lucene.php(76): index()\n#4 \/var\/www\/owncloud\/lib\/base.php(798): require_once('\/var\/www\/ownclo...')\n#5 [internal function]: OC::loadAppScriptFile(Array)\n#6 \/var\/www\/owncloud\/lib\/private\/route\/router.php(227): call_user_func(Array, Array)\n#7 \/var\/www\/owncloud\/lib\/base.php(706): OC\\Route\\Router->match('\/apps\/search_lu...')\n#8 \/var\/www\/owncloud\/index.php(28): OC::handleRequest()\n#9 {main}","level":3,"time":"2015-01-11T10:13:53+00:00","method":"GET","url":"\/owncloud\/index.php\/apps\/search_lucene\/ajax\/lucene.php?operation=index&requesttoken=5b86f79389f5eff37b8e"}
{"reqId":"54b24ce185af7","app":"PHP","message":"Allowed memory size of 536870912 bytes exhausted (tried to allocate 4248 bytes) at \/var\/www\/owncloud\/lib\/autoloader.php#142","level":3,"time":"2015-01-11T10:13:53+00:00","method":"GET","url":"\/owncloud\/index.php\/apps\/search_lucene\/ajax\/lucene.php?operation=index&requesttoken=5b86f79389f5eff37b8e"}
{"reqId":"54b24d2fe6768","app":"PHP","message":"Allowed memory size of 536870912 bytes exhausted (tried to allocate 72 bytes) at \/var\/www\/owncloud\/3rdparty\/doctrine\/dbal\/lib\/Doctrine\/DBAL\/Statement.php#216","level":3,"time":"2015-01-11T10:15:11+00:00","method":"GET","url":"\/owncloud\/index.php\/apps\/search_lucene\/ajax\/lucene.php?operation=index&requesttoken=5b86f79389f5eff37b8e"}

after removing all *.xlsx files the error does not occur any more (but this is not a solution)

Browser log

Insert your browser log here, this could for example include:

a) The javascript console log
b) The network log 
c) ...

[VicDeo]

@butonic @DeepDiver1975
I guess we need index N files per iteration. Let's say 20 files in one go. What do you think?

[butonic]

indexing should become a background job only. we can then index all files in the background. the admin settings page can then show the number of currently indexed files. Note that before that we should move to a single index, see #10

[DeepDiver1975]

Note that before that we should move to a single index, see #10

didn't we have some serious concerns with respect to use a single index? Maybe I miss something ...

[DeepDiver1975]

indexing should become a background job only.

this is already the case - or do I miss anything?

[VicDeo]

@DeepDiver1975 no, it starts indexing when search box is clicked too https://github.com/owncloud/search_lucene/blob/master/js/checker.js#L45

[VicDeo]

@DeepDiver1975 @butonic let's discuss single index further in #10
I interested in the concerns behind it