Set iterator read bounds on deleting column families to avoid very slow syncs

[teor2345]

Motivation

Iterating on a column family that deletes lots of keys can be very slow.

We fixed some of this code in PRs #7663 and #7392, but we need a general fix to stop this issue happening again.

This currently impacts the utxo_loc_by_transparent_addr_loc column family, and any future uses of iterators with zs_delete(). It can be remotely triggered.

Specifications

The RocksDB documentation recommends specifying lower and upper bounds on iterators:
https://github.com/facebook/rocksdb/wiki/Iterator#iterating-upper-bound-and-lower-bound
https://docs.rs/rocksdb/latest/rocksdb/struct.ReadOptions.html#method.set_iterate_range
https://docs.rs/rocksdb/latest/rocksdb/struct.DBCommon.html#method.iterator_cf_opt

This is how these bugs in other projects...
https://tracker.ceph.com/issues/55324
https://jira.mariadb.org/browse/MDEV-19670

Were fixed in those projects:
https://github.com/ceph/ceph/pull/46096/files
facebook/rocksdb#5403

Complex Code or Requirements

We need to use iterators for some operations, so our alternatives are (in preferred order):

1. Minimise the number of keys we delete, and how often we delete them (currently only UTXOs require key deletion)
2. Avoid using iterators on column families where we delete keys (currently only one column family does this, and it won't be necessary after PR fix(security): fix concurrency issues in tree key formats, and CPU usage in genesis tree roots #7392)
3. If we must use iterators on those column families, set read bounds to minimise the amount of deleted data that is read (this ticket)

If this solution passes our benchmark tests, we should document it in:
https://github.com/ZcashFoundation/zebra/blob/main/book/src/dev/state-db-upgrades.md

Testing

When we implement the test in ticket #7649, it will also cover this change.

Documentation

We should document this issue in the developer state upgrade docs.

Related Work

This is a long-term fix for #7618.

PR #7663 implements alternative 2 for some column families.
PR #7392 implements alternative 1 for some column families.

[mpguerra]

Hey team! Please add your planning poker estimate with Zenhub @arya2 @oxarbitrage @teor2345 @upbqdn

[teor2345]

This could be useful for making an exclusive end bound:
https://docs.rs/primitive-types/0.12.2/primitive_types/struct.U256.html#method.checked_add

[teor2345]

@arya2 I just added this to the ticket, I'm happy to do it (along with a bunch of other state docs):

Documentation

We should document this issue in the developer state upgrade docs.

[mpguerra]

@arya2 and @teor2345 who is taking the lead on this one?

[arya2]

who is taking the lead on this one?

Teor and I talked through what changes are needed, and I'll be writing them up.