Benchmarking

[alfonsorr]

@zeotuan gave the idea of benchmarking the library. It is a good way to compare different implementations and see how any changes affect performance.
I suggest creating a module in this project that contains all the code for the benchmark. That way we won't have to be taking tests, and it also makes it easier to skip them in some CI steps if needed.

@zeotuan suggested https://github.com/sbt/sbt-jmh . I'm not an expert in this field but looks well maintained so far.

Also, it will be great to find a good data file to use as a reference. Also see the limitations of github actions if we are going to run it there as test hardware.

[zeotuan]

@alfonsorr what do you think about using a multi-project structure?

/project-root
  ├── benchmarks
  │   ├── src
  │   │   ├── main
  │   │   │   └── scala
  │   │   │       └── com.github.mrpowers.spark.fast.benchmarks
  │   │   │           └── ...
  ├── core
  │   ├── src
  │   │   ├── main
  │   │   │   └── scala
  │   │   │       └── com.github.mrpowers.spark.fast.tests
  │   │   │           └── ...
  │   │   └── test
  │   │       └── scala
  │   │           └── com.github.mrpowers.spark.fast.tests
  │   │               └──...
  └── build.sbt

one is core which contain all the original code
The other one is benchmarks with all the benchmark code we can also defined it to depends on core in build.sbt.
benchmark can then use additional dependency such as data generation library without affecting core