-
Notifications
You must be signed in to change notification settings - Fork 177
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add support for pooling in codecs, partition_table, view etc. #445
base: master
Are you sure you want to change the base?
Conversation
87c2b2c
to
d7194bc
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Therefore we need to be able to tell the Codec when the message processing is done and the reference can be reused.
So codecs become stateful? Or were there already before?
codec/closer.go
Outdated
type nullCloser struct{} | ||
|
||
func (n *nullCloser) Close() error { return nil } | ||
|
||
var NoopCloser = new(nullCloser) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What is this for, if all the code calling DecodeP
is doing a nil
check on the io.Closer
return.
Thanks @paulhenke for all your comments. As said in some comments, the idea of this PR is to draft how pooling could be supported, although goka does not actually use it, only the clients. The codecs would become stateful in some sense, that's true, but they haven't been before and don't have to be. But breaking the API for this is probably a bad idea, so we'll design it in a way that both ways are supported. |
d6a8f5a
to
c1f7942
Compare
Work in Progress
This is a draft for evaluating the efficiency and consequences of adding (memory) pooling to the goka API.
How it works
When processing messages with goka
Processor
s, messages are decoded using aCodec
in every callback call, as well as for everyJoin
orLookup
. The same happens on everyView.Get
. When using more complex codecs like protobuf, those allocations add up and create pressure in the garbage collector. We would like to try reusing the messages. Protobuf supportsReset
ing the message upon unmarshalling, so they're a god fit for pooling.Therefore we need to be able to tell the
Codec
when the message processing is done and the reference can be reused.This PR adds a new decode-function called
DecodeP
which returns anio.Closer
interface that must be called when the message can be reused.Additionally, it also adds a
GetP
method to thestorage.Storage
interface to support pooling on disk-level, especially for Views.