Questions about memory usage

[quentin-bettoum]

Hello,

I am bulding a webapp where I use Vips to thumbnail some uploaded images.
I noticed that when the images are being thumbnailed, the memory usage of the beam.smp process is growing significantly and never go down to original values.
I could replicate the issue in a simple project: https://codeberg.org/quentin-bettoum/test_vix_memory

There is some discrepancy between what the OS reports and what is reported from Elixir using :recon or other tools.
When thumbnailing images, the beam.smp process RAM usage grows from around 90M to somewhere between 200M and 300M, and it doesn't go down anymore after that (even when trying to manually garbage collect calling :erlang.garbage_collect()).

But when I use :recon or Phoenix Live Dashboard (in case of the web app) I can't see any memory usage spike as with system tools like top.

Any help to understand what is happening would be appreciated.

[akash-akya]

@quentin-bettoum libvips (underlying library which vix uses) allocates memory separately which won't be tracked by the beam. So that is the reason for the discrepancies you are seeing, and that's why it won't be visible to the tools like recon

libvips keeps a cache by default for better performance, that might be the reason for the memory usage you are seeing. Typically this won't be an issue. Can you check and confirm if you see any difference when you disable cache? You can do this by setting them to zero at the top of the script, before calling the operations.

Vix.Vips.cache_set_max(0) 
Vix.Vips.cache_set_max_files(0)
Vix.Vips.cache_set_max_mem(0)

If you still see the issue then I'll debug with the script you shared as soon as I get some free time.

[quentin-bettoum]

Thanks for your answer,

I can see a difference, now it still goes from 90M to above 200M, but then it goes down to around 165M (sometimes after a few seconds).

It still wont go back near the inital value, but it's a lot better!

I also found that setting VIPS_CONCURRENCY to 1 (or via Vix.Vips.concurrency_set(1)) will reduce spikes.