Add id column as PRIMARY KEY for evm.logs & evm.log_poller_blocks #1441
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
reductionista
commented
Sep 16, 2024
| @@ -1565,7 +1565,7 @@ func TestSelectLatestBlockNumberEventSigsAddrsWithConfs(t *testing.T) { | |||
| events: []common.Hash{event1, event2}, | |||
| addrs: []common.Address{address1, address2}, | |||
| confs: 0, | |||
| fromBlock: 3, | |||
There was a problem hiding this comment.
This behavior looked wrong to me, hopefully there aren't any upstream clients depending on it:
everywhere else, passing a fromBlock implies you're searching for blocks >= fromBlock. This code has been updated in this PR to use that now instead of matching only on blocks > fromBlock
reductionista
temporarily deployed
to
publish
GitHub Actions
Inactive
reductionista
force-pushed
the
log_poller_id_columns
branch
from
c03e447
to
0d1c216
Compare
reductionista
temporarily deployed
to
publish
GitHub Actions
Inactive
reductionista
temporarily deployed
to
publish
GitHub Actions
Inactive
reductionista
had a problem deploying
to
publish
GitHub Actions
Error
reductionista
force-pushed
the
log_poller_id_columns
branch
from
b20e122
to
52d080c
Compare
reductionista
temporarily deployed
to
publish
GitHub Actions
Inactive
reductionista
force-pushed
the
log_poller_id_columns
branch
from
52d080c
to
069660e
Compare
reductionista
temporarily deployed
to
publish
GitHub Actions
Inactive
reductionista
temporarily deployed
to
publish
GitHub Actions
Inactive
BCI-3492 [LogPoller]: Allow withObservedExecAndRowsAffected to report non-zero rows affected (#14057)
* Fix withObservedExecAndRowsAffected
Also:
- Change behavior of DeleteExpiredLogs to delete logs which don't match any filter
- Add a test case to ensure the dataset size is published properly during pruning
* pnpm changeset
* changeset #fix -> #bugfix
reductionista
force-pushed
the
log_poller_id_columns
branch
from
94e15fa
to
a99e37f
Compare
reductionista
force-pushed
the
log_poller_id_columns
branch
from
a99e37f
to
c6c76ae
Compare
commit 2761cd5 Author: Juan Farber <[email protected]> Date: Wed Sep 4 17:16:00 2024 -0300 [BCI-3988] - FilteredLogs receive []Expression instead of whole KeyFilter (#14109) * FilteredLogs receive []Expression instead of whole KeyFilter * remove key from query.Where KeyFilter creation * add changeset * remove brackets from changeset * fix usage * fix comment lint * remove todo * refactor based on comments * add where func inside logpoller without key * fix reference
Also:
- Add UNIQUE INDEXes to replace previous primary keys (still necessary, both for
optimizing queries and for enforcing uniqueness constraints)
- Replace all SELECT *'s with helper functions for selecting all columns
- Refactor nestedBlockQuery into withConfs, and make a bit more use of it
Some of the columns in these indexes (such as created_at) are no longer used. Others were not optimized for the queries we need.
Previously it was using fromBlock > :block_number which is inconsistent with the other fromBlocks in queries
On a node with more than one chain, each LogPoller would have deleted all logs from chains it's not running on! Because of the LEFT JOIN, ON evm_chain_id = $1 does not filter out any rows where evm_chain_id != $1; only WHERE evm_chain_id = $1 can do that
reductionista
force-pushed
the
log_poller_id_columns
branch
from
cde8d5e
to
ae7d150
Compare
reductionista
had a problem deploying
to
sdlc
GitHub Actions
Failure
reductionista
had a problem deploying
to
sdlc
GitHub Actions
Failure
reductionista
had a problem deploying
to
publish
GitHub Actions
Failure
reductionista
temporarily deployed
to
publish
GitHub Actions
Inactive
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Problem:
The query for pruning expired logs with a max limit set was taking longer than it should. This was part due to needing to join on an awkward combination combination of columns due to their being no single primary key.
Solution:
While adding the id column, we can't just remove the old primary key because the index on it was helping to accelerate some queries.
Instead of just resurrecting it as-is, I took the opportunity to clean up several of the indices on the logs & blocks table. Some indexed columns (eg created_at) were never actually being used,
while others were not ordered in the most optimal way for accelerating the queries we have. Also, at least one of them was redundant with the primary key just in a different order.