Update Library database scans are extremely slow

[sleeve]

Updating a large .rar video library database with 2.2.2 is taking an extremely long time compared to 2.0.8.

I have a shared kodi mysql database running on a local server that multiple clients connect to. I usually update my video library manually on my fastest machine which is running Windows 10. With a large .rar video collection of 2000+ directories the update library process takes about 5 minutes with vfs.rar-2.0.8. Running the same update library scan with vfs.rar-2.2.2 takes about 1 hour and 20 minutes!

[AlwinEsch]

Think this fix it a2ab4ba

Found by a music RAR it takes extremely long and Kodi also Freeze during his stop and has needed a killall -9 kodi-x11 😏.

[sleeve]

Thanks, I'm happy to test a fix if it will be included in a Leia build.

[AlwinEsch]

Have created here a request about Leia #77, is still WIP and maybe Problems on Windows (slash and back slash thing).

Here are binaries available: https://jenkins.kodi.tv/blue/organizations/jenkins/xbmc%2Fvfs.rar/detail/PR-77/1/artifacts

[AlwinEsch]

There is now a newer, where the slash problem hopefully fixed:
Leia: https://jenkins.kodi.tv/blue/organizations/jenkins/xbmc%2Fvfs.rar/detail/PR-77/3/artifacts
Matrix: https://jenkins.kodi.tv/blue/organizations/jenkins/xbmc%2Fvfs.rar/detail/PR-75/8/artifacts

[AlwinEsch]

The new version is released:

• 3.3.0-Matrix
• 2.3.0-Leia

Can you check if better now?

[sleeve]

Thanks, I should have time within the next day or so to run more tests. Thanks for all the continued work to get all these issues fixed!

[sleeve]

Sorry for the delay on getting this tested. This is still on my todo list but I unfortunately need to rebuild some servers before I can run a full test.

[bluboyo]

My scans are very slow to. 18.7, 2.3.1. Big dif compeard to 2.0.8

[sleeve]

Apologies again for the delay in testing but I was just able to run another test comparing 2.0.8 performance against 2.3.1 and I'm still seeing similar results. I was able to find the exact time taken for a full library scan in the log file. Here are the results.

2.0.8

NOTICE: VideoInfoScanner: Finished scan. Scanning for video info took 05:27

2.3.1

NOTICE: VideoInfoScanner: Finished scan. Scanning for video info took 01:35:35

So unfortunately it looks like the speculative fix didn't work.

[sleeve]

@AlwinEsch would you mind removing the bug fixed tag for this issue so it can still be investigated. This is the only issue that has kept me from upgrading from 2.0.8.

[kraaa]

I don't understand why Kodi rescans all rar files once they have already been added and scanned before.
Should it not be sufficed to just scan the directory for an old single .rar file for example to make sure it is there?
Hoping for a solution.

As an example one of my debug logs is 77mb and then it crashed on a .sub file.

[Gomez]

I have a similar problem. Looks like a scan extracts all rars (or some) into the vfs.rar-temp dir:

I had to cleanup as my disk runs full.

drwx------    2 root     root        4.0K Sep 24 14:24 rarfolder6eKrzk
drwx------    2 root     root        4.0K Sep 24 13:47 rarfoldereiN5CA
drwx------    2 root     root        4.0K Sep 24 13:47 rarfolderqD0M49
drwx------    2 root     root        4.0K Sep 24 13:45 rarfoldere6ityK
drwx------    2 root     root        4.0K Sep 24 13:42 rarfolderMgz8fO
drwx------    2 root     root        4.0K Sep 24 13:30 rarfolderAJvaZN
drwx------    2 root     root        4.0K Sep 24 13:29 rarfolderwKNAb7
drwx------    2 root     root        4.0K Sep 24 13:27 rarfolderQVALgx
drwx------    2 root     root        4.0K Sep 24 12:28 rarfolderkCqpD2
drwx------    2 root     root        4.0K Sep 24 12:28 rarfolderU7A5Z1
drwx------    2 root     root        4.0K Sep 24 12:26 rarfolders6VmT3
drwx------    2 root     root        4.0K Sep 24 12:25 rarfolderIbFKo8
drwx------    2 root     root        4.0K Sep 24 12:05 rarfolder01LnGy
drwx------    2 root     root        4.0K Sep 24 12:04 rarfoldercBcL9P
drwx------    2 root     root        4.0K Sep 24 11:07 rarfoldere30Cci
drwx------    2 root     root        4.0K Sep 24 10:42 rarfolderA6Qds4
drwx------    2 root     root        4.0K Sep 24 10:23 rarfolderGknoAe
drwx------    2 root     root        4.0K Sep 24 10:00 rarfolderDTNwjd
drwx------    2 root     root        4.0K Sep 24 09:42 rarfoldercONng6
drwx------    2 root     root        4.0K Sep 24 09:27 rarfolderrvq84j
drwx------    2 root     root        4.0K Sep 24 09:26 rarfolder8btKQy
drwx------    2 root     root        4.0K Sep 24 09:26 rarfolder3p6KTR
drwx------    2 root     root        4.0K Sep 24 09:25 rarfolderQJYFHb
drwx------    2 root     root        4.0K Sep 24 09:22 rarfoldernjcfOe
drwx------    2 root     root        4.0K Sep 24 09:13 rarfolderavLGd7
drwx------    2 root     root        4.0K Sep 24 09:12 rarfoldernwl01n
drwx------    2 root     root        4.0K Sep 24 09:11 rarfolderwRGrhX
drwx------    2 root     root        4.0K Sep 24 08:30 rarfolderR0KaQS
drwx------    2 root     root        4.0K Sep 24 08:30 rarfolderat78sW
drwx------    2 root     root        4.0K Sep 24 08:29 rarfoldervCpUx1
drwx------    2 root     root        4.0K Sep 24 08:27 rarfolderqKE7UA
drwx------    2 root     root        4.0K Sep 24 07:59 rarfolderXXqUcw
drwx------    2 root     root        4.0K Sep 24 07:58 rarfolderGFzULo
drwx------    2 root     root        4.0K Sep 24 07:10 rarfolderTOvK8i
drwx------    2 root     root        4.0K Sep 23 22:09 rarfolderzYhtbO
drwx------    2 root     root        4.0K Sep 23 21:30 rarfolderBcnum4
drwx------    2 root     root        4.0K Sep 23 21:13 rarfolderGwMa68
drwx------    2 root     root        4.0K Sep 23 20:57 rarfolder04MyQP
drwx------    2 root     root        4.0K Sep 23 20:32 rarfoldern6bgAZ
drwx------    2 root     root        4.0K Sep 23 20:15 rarfolderehYQYh
drwx------    2 root     root        4.0K Sep 23 20:04 rarfoldery2ExQD
drwx------    2 root     root        4.0K Sep 23 20:03 rarfolder5odnMO
drwx------    2 root     root        4.0K Sep 23 19:10 rarfolderkCO23Z
drwx------    2 root     root        4.0K Sep 23 14:42 rarfolderQ4gPMT
drwx------    2 root     root        4.0K Sep 23 14:39 rarfolderr3Qbhh
drwx------    2 root     root        4.0K Sep 23 14:38 rarfolder21FXnq
drwx------    2 root     root        4.0K Sep 23 14:32 rarfolderJzHiyp
drwx------    2 root     root        4.0K Sep 23 13:49 rarfolder4TadrS
drwx------    2 root     root        4.0K Sep 23 13:17 rarfolderVYuxD8
drwx------    2 root     root        4.0K Sep 23 13:17 rarfolderikjqnc
drwx------    2 root     root        4.0K Sep 23 13:16 rarfolderjENn6g
drwx------    2 root     root        4.0K Sep 23 13:13 rarfolderCRLsku
drwx------    2 root     root        4.0K Sep 23 13:03 rarfolderlyIX2G
drwx------    2 root     root        4.0K Sep 23 13:02 rarfolderqk2GSH
drwx------    2 root     root        4.0K Sep 23 13:01 rarfolderFN3Ld3
drwx------    2 root     root        4.0K Sep 23 12:27 rarfolderYuZTLP
drwx------    2 root     root        4.0K Sep 23 12:27 rarfolderjtQdX0
drwx------    2 root     root        4.0K Sep 23 12:25 rarfoldercnpOqd
drwx------    2 root     root        4.0K Sep 23 12:24 rarfolderTSjZPS
drwx------    2 root     root        4.0K Sep 23 12:04 rarfolderMQIPRY
drwx------    2 root     root        4.0K Sep 23 11:59 rarfolderZIGydc
drwx------    2 root     root        4.0K Sep 23 11:18 rarfolder0cidIb
drwx------    2 root     root        4.0K Sep 23 10:55 rarfolderHVC8e7
drwx------    2 root     root        4.0K Sep 23 10:38 rarfoldereXQCBc
drwx------    2 root     root        4.0K Sep 23 10:22 rarfolderU7pcXJ
drwx------    2 root     root        4.0K Sep 23 10:15 rarfolderbH3PrE
drwx------    2 root     root        4.0K Sep 23 09:33 rarfolderuryx0G
drwx------    2 root     root        4.0K Sep 22 19:40 rarfoldercF0ujm
drwxr-xr-x    3 root     root        4.0K Sep 12 19:00 ..             
LibreELEC:~/.kodi/temp/binary-addons/vfs.rar-temp #  

[n0rdik]

+1. I almost run out of disk space and it takes around 30 minutes to perform the scan.

[Gomez]

+1. I almost run out of disk space and it takes around 30 minutes to perform the scan.

I disabled the scan for now...

[biner73]

I've noticed the same with anything 2.0.8+ on Leia, as well as 4.0.0 on the latest beta of Matrix, taking forever to update the library. 2.0.8 on Leia takes seconds.

[ZevZev]

For me: in Kodi 18 with vfs.rar >2.0.8: a lot of different issues.
I kept using 2.0.8 without any issues.

Now in Kodi 19 different issues. If I install both Libarchive + vfs.rar - then scanning takes forever as posted in this thread.
If I remove libarchive and only use vfs.rar the scanning speed is ok again - but there are other issues:
#102