Re-write or remove the use of CACHELINE

[lundgren87]

CACHELINE is a hard-coded parameter in the MPI backend that can be roughly translated to number of pages (of size PAGE_SIZE) per ArgoDSM page for both cache-related operations and Pyxis (coherence) operations. This is currently untested due to being hard-coded, and I am certain that this does not work with all of our code, allocation policies in particular.

argodsm/src/backend/mpi/swdsm.h

Lines 45 to 48 in 5a6d253

	#ifndef CACHELINE
	/** @brief Size of a ArgoDSM cacheline in number of pages */
	#define CACHELINE 1L
	#endif

I think the intended effects of CACHELINE are desirable in the sense of a tuning knob to trade false sharing for a reduction in remote operation. For workloads that access contiguous, large (in the sense of several pages or more), data appropriately divided between nodes, there is no such "false sharing". Using a larger PAGE_SIZE would for such workloads reduce the amount of remote operations without any significant drawbacks.

A proper implementation of this should not rely on repeatedly incorporating PAGE_SIZE*CACHELINE in the backend for all cache or coherence operations. One solution is to instead use an environment variable to set the page size in a controlled way (for example in powers of two hardware pages). Most, if not all of the calculations based on PAGE_SIZE should then be easily adaptable to a larger page size even in the current backend implementation.

Another alternative is to simply remove the CACHELINE parameter as it is currently of no use.