Refactor benchmarks

[oruebel]

Refactor benchmarks to separate the definition of test cases from the specific metrics. The idea is to define the test case in a base class and then the actual benchmark class defines the metric to be run for the test cases. The goal is to make it easy to run the same test with different metrics without having to repeat the implementation of the test case.

[oruebel]

@CodyCBakerPhD what do you think about a design like this for the tests? The main idea is that I would like to be able to compute multiple metrics for the same test case, but I would like to avoid having to copy-paste the same case multiple times just to change the metric.

[CodyCBakerPhD]

Overall definitely a good idea; left some suggestions for improving the hierarchy

We could even go down the road of having 'mixin' tests if we want to define common types that ought to apply with mix and matchable setups

[oruebel]

left some suggestions for improving the hierarchy

Thanks @CodyCBakerPhD for the helpful suggestions. I made some more changes to implement the changes you had suggested. Can you please take another look.

[CodyCBakerPhD]

@oruebel Thanks for working with me on this; I've opened #19, #20, and #21 in draft mode as demonstrations of alternative strategies

We should really put some thorough thought into what will make it easiest for us to both (a) write tests quickly (as in, now, during development) without as you say, copying/pasting too much, as well as (b) for us to read (in the far future) to quickly remember and understand what is being setup and tested in each case

The earlier we decide on a refactor is put in place, the less work it will be compared to fixing it down the line

Too much implicit structure and redirection can hamper (b), but we need some amount to speed up and (a) and keep the main benchmarks relatively clean, so a balance must be struck somehow

For example, on this PR, if we wanted to answer the question 'what is FileReadStreaming testing?'

We can see quickly that it runs 3 time tests that redirect to class methods. We follow the parent to the streaming base to discover the method definitions in each case, so this one is not too difficult to go back and read.

However, if we wanted to answer the question 'what is ElectricalSeriesStreamingFsspecBase testing?'

We can see that a single time test is being run, and is running a slice request. But this time we have to follow the parent to find the setup definition to know/check the protocol is indeed fsspec-based, then follow its parent to find the slice request definition (note none of this is actually a mixin strictly speaking, just pure inheritance)

Also note that in both cases, for development, if we wanted to make a slight change to the fsspec read protocol, we'd have to remember to modify it in both the FileReadStreamingBase and the ElectricalSeriesStreamingFsspecBase so ensure a fair comparison and synchrony between the two cases; this could have easily been missed because the read method isn't entirely centralized

As an aside I realized the context method is also going to track the __exit__ conditions and ensuing garbage collection

[oruebel]

Thanks for working with me on this; I've opened #19, #20, and #21 in draft mode as demonstrations of alternative strategies

We should really put some thorough thought into what will make it easiest for us to both (a) write tests quickly (as in, now, during development) without as you say, copying/pasting too much, as well as (b) for us to read (in the far future) to quickly remember and understand what is being setup and tested in each case

I totally agree. I appreciate that you are taking such a thoughtful approach to this. I think it would be easiest for us to review the different proposed variants via Zoom so that we can decide on a strategy.

[CodyCBakerPhD]

replaced by #21