PartitionStoreManager allows to drop a partition while PartitionStores are still active

[tillrohrmann]

At the moment it is possible to drop a the CF of a partition via PartitionStoreManager::drop_partition while other components still hold the corresponding PartitionStore to access the data. At the moment we rely on RocksDB to handle this situation correctly (which it presumably does as testing shows so far).

Currently, the LocalPartitionsScanner can access the PartitionStore outside of the PartitionProcessor and its runtime. This might lead to accesses while the underlying partition CF is being dropped.

A more robust design would be to invalidate the PartitionStore when dropping the corresponding partition or to only allow the partition to be dropped if all PartitionStores are closed somehow. Alternatively, we can rethink whether dropping the partition CF concurrently is strictly required.

[igalshilman]

To keep the ideas discussed offline in the same issue:

Another approach is to make sure that there is a clear ownership of the partition data, by the partition processes manager that drives the lifecycle of the partition. Any task that needs to interact with the partition data needs to be submitting to its runtime.
Before disposing the partition data, it needs to verify that the tasks won't run anymore.