Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Increase checkpoint block number when there are no events, to minimize cost on restart #105

Open
peterbroadhurst opened this issue Jan 21, 2021 · 0 comments
Assignees

Comments

@peterbroadhurst
Copy link
Contributor

Currently we only increase the block high-watermark on a subscription to an event stream, when we have successfully processed a batch:
https://github.com/kaleido-io/ethconnect/blob/ad40de61370ec2b99a8cfc79cdf72e5dcac3d914/internal/kldevents/logprocessor.go#L83

This means that if no events arrive on a subscription for a long time (millions of blocks) we never increase the blockHWM, and write a new checkpoint. So on restart (of ethconnect or the node), we drive a high load on the node as we recreate the filter at a very old block number.

Instead we need a system to safely move the blockHWM forwards, at regular intervals, so it's never more than a few hundred blocks behind the head. This will mean periodically in the polling cycle, asking the node for the current block height ahead of the call to eth_getFilterChanges and if no results are found, then setting the blockHWM to a safe number (such as 100 blocks behind the head, if that's higher than zero, and higher than the previous blockHWM).

fyi @vdamle @panghalamit

@peterbroadhurst peterbroadhurst self-assigned this Jan 21, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant