Memory leak detected.

[pmiotto]

Hi.
It seems that when uploading an image with multiple resizes causes memory leaks under a small load. I've performed a load test using this library and it fails around 15 concurrent requests. Here's the error stack. Any help would be appreciated.

MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 error listeners added. Use emitter.setMaxListeners() to increase limit
at _addListener (events.js:247:17)
at Sharp.addListener (events.js:263:10)
at Sharp.Readable.on (_stream_readable.js:800:35)
at ManagedUpload.send (/Users/me/Documents/wip/project/node_modules/aws-sdk/lib/s3/managed_upload.js:185:11)
at /Users/me/Documents/wip/project/node_modules/aws-sdk/lib/util.js:798:25
at new WrappedPromise (/Users/me/Documents/wip/project/node_modules/async-listener/es6-wrapped-promise.js:13:18)
at ManagedUpload.promise (/Users/me/Documents/wip/project/node_modules/aws-sdk/lib/util.js:797:14)
at MergeMapSubscriber.sizes.pipe.operators_1.mergeMap [as project] (/Users/me/Documents/wip/project/node_modules/multer-sharp-s3/dist/main.js:113:52)
at MergeMapSubscriber._tryNext (/Users/me/Documents/wip/project/node_modules/rxjs/internal/operators/mergeMap.js:69:27)
at MergeMapSubscriber._next (/Users/me/Documents/wip/project/node_modules/rxjs/internal/operators/mergeMap.js:59:18)
at MergeMapSubscriber.Subscriber.next (/Users/me/Documents/wip/project/node_modules/rxjs/internal/Subscriber.js:66:18)
at MergeMapSubscriber.notifyNext (/Users/me/Documents/wip/project/node_modules/rxjs/internal/operators/mergeMap.js:92:26)
at InnerSubscriber._next (/Users/me/Documents/wip/project/node_modules/rxjs/internal/InnerSubscriber.js:28:21)
at InnerSubscriber.Subscriber.next (/Users/me/Documents/wip/project/node_modules/rxjs/internal/Subscriber.js:66:18)
at /Users/me/Documents/wip/project/node_modules/rxjs/internal/util/subscribeToPromise.js:7:24
at propagateAslWrapper (/Users/me/Documents/wip/project/node_modules/async-listener/index.js:504:23)
at /Users/me/Documents/wip/project/node_modules/async-listener/glue.js:188:31
at /Users/me/Documents/wip/project/node_modules/async-listener/index.js:541:70
at /Users/me/Documents/wip/project/node_modules/async-listener/glue.js:188:31

[ikhsanalatsary]

Hi @pmiotto thanks for reporting this. how to reproduce like that?

[pmiotto]

HI @ikhsanalatsary. I've got an endpoint which uploads an image and a couple of resizes to S3, you can use a tool like jMeter to perform a loadtest. It fails under 10 concurrent requests. Not only I get this memory leak but some images uploaded to S3 are 0kb.

[ikhsanalatsary]

@pmiotto could you share your JMeter configuration?

[ikhsanalatsary]

My configuration

and sometimes one image is still uploaded 0 Bytes but at least it reduces the total from previous.
So, You can try again in this branch https://github.com/ikhsanalatsary/multer-sharp-s3/tree/fix/bumps-version
PS: I need feedback. thanks

[mohmmadmoussa1988]

Hello Dear,

Any solution for this case ?

[Milad]

This is expected to happen because you have many unused variables. Start by eliminating the unused variables from your code.

[djuko]

I am also having this issue. It is because I am having a long resize set. Is there a way to do this in a way of asynchronous calls (Promisse.all or something like that) to relieve the EventEmitter ?

[djuko]

I have increased for now the event emitter with require('events').EventEmitter.defaultMaxListeners = 40, but this should be heavily avoided. I suggest we find a way to implement async await when uploading multiple images. I'll follow up on this if I find a way and let you guys know :)

[Milad]

Of course it can be promisified easily, and if not, then it should!

[djuko]

I agree it should be promisified. I'll take a look at the src code after working hours, it would be best to implement it there