You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The RunLogger class caches the Output event and writes it to a local variable (node_output). There are two issues with the current implementation which might cause high memory usage during pipeline execution:
when you have a long running task / parallel task, a long output chain might be cached. There should be a solution implemented which writes the cached output to the db after e.g. 500 entries.
the node_output dict is not freed after writing the output to the dabase. This will result in that the whole pipeline ouput is cached in RAM until class RunLogger is disposed (!!)
The text was updated successfully, but these errors were encountered:
leo-schick
changed the title
Memory issue with RunLogger
High memory issue with RunLogger
May 18, 2022
I noted that the ChatNotifier has a similar issue: It fees the node_output only on RunFinished, not on NodeFinished but this might be wanted. This is out of scope for this ticket.
The
RunLogger
class caches the Output event and writes it to a local variable (node_output). There are two issues with the current implementation which might cause high memory usage during pipeline execution:node_output
dict is not freed after writing the output to the dabase. This will result in that the whole pipeline ouput is cached in RAM until class RunLogger is disposed (!!)The text was updated successfully, but these errors were encountered: