Investigate performance impact of enabling and using PSHM with GASNet

[bradcray]

When running multiple co-locales per node, there is an opportunity to have co-located co-locales communicate using shared memory. When using CHPL_COMM=gasnet this can be done using GASNet's support for inter-Process SHared Memory, or PSHM. This issue captures our desire to understand the performance differences for a swath of benchmarks running on multiple nodes when using GASNet in a number of configurations:

1D) running 1 locale per node ("classic Chapel")
1E) running 1 locale per node with PSHM enabled at build-time, but unused
2D) running 2 co-locales per node with PSHM disabled
2ED) running 2 co-locales per node with PSHM enabled at build-time, but not used at execution time
2EE) running 2 co-locales per node with PSHM enabled and used

The goal being to understand cases where enabling and/or using PSHM hurts performance relative to not, and with the ultimate goal of enabling PSHM by default when using co-locales.

The ideal would be for 1E to not hurt performance relative to 1D and for 2ED to not hurt performance relative to 2D, which would suggest that PSHM could always be enabled at build time. In addition, if 2EE always matched or beat 2ED, that would suggest that PSHM should always be used by default when using co-locales. In practice, if both these things were true, Chapel users would operate in 1E or 2EE unless they took pains to disable something.

[bradcray]

Note that we are currently gathering numbers for 1D, 2ED, and 2DD at https://chapel-lang.org/perf/16-node-apollo-hdr/colocales but that these numbers come with the caveat that most of our performance tests were written prior to co-locales coming into existence, and so the things they report and measure may not be directly comparable without additional changes to the benchmarks or interpretation of the results.

While perusing these graphs recently, I grabbed the following links to note:

• the subset of graphs for which using co-locales behaves notably differently from using a single locale per node (i.e., 1D differs from 2E*)
• the subset of graphs for which using PSHM changes the co-locale performance (i.e., 2ED differs from 2EE)

(where I tried to err on the side of including graphs that seemed to show differences since it's easier to drop graphs from the set rather than adding new ones in)