automate performance tuning

[vadasg]

For optimal MapReduce performance, a sequence file should be generated and there are several parameters that must be tuned. This process could be automated:

1. Start with some default set of values for parameters like

mapred.map.child.java.opts
mapred.reduce.child.java.opts
mapred.map.tasks
mapred.reduce.tasks
etc

2. Attempt to generate sequence file with g.(). Determine size of biggest vertex in memory during this process. If a vertex doesn't fit in memory (so g.() fails), increase memory automatically and retry until it does. If it fails for other reasons, iteratively change above parameters until it succeeds. Restart regionservers automatically as needed.

3. Once sequence file is generated, determine optimal parameters based on the graph and based on the available memory and number of processors. Generate a parameter file for future analyses.

The idea is to have a possibly long but automated process that will eventually converge on reasonably well optimized parameters.