Feedback on kernels

[msaroufim]

From Erik and Matej

• conv: spec should be part of input_t - make sure input, padding and striding is passed to eval. Get rid of stride and padding
• if new input gets added make sure its input_spec and also returned by generate_input()
• Promise that test cases are powers of 2
• If it takes less than a 1ms to run then make it bigger
• Make the promises explicit in the problem descrition yml
• Test spec should be a single mega object always
• Histogram testing is interesting: what input distributions, every number is the same otherwise be explicit about assumptions of the input
• Have input tests give multiple input distributions, add another arugment to the testspec, large inputs means low bit dypes will be f'd and lean on input distrubtions where the numbers are small so maybe be explicity around thei nput distributions as well
• Sorting: we need multiple distributions, already sorted, inversely sorted, random, large, small, etc.
• Problem reuse is important because we can be more specific about distributions or hard code sizes, square for now but hey maybe rectangular later
• Maybe hacking is good? Because we can monitor issues
• Basically you have to make more assumptions than PyTorch to beat it