Start-up time is included in Mason benchmarks?

[Tortar]

Looking closely at the results of the benchmarks for Mason, we can see that some start-up time is biasing the results:

• Flocking small in Agents.jl takes ~10 ms while in Mason it takes ~200 ms
• Flocking large in Agents.jl takes ~90 ms while in Mason it takes ~280 ms

Clearly there is something odd going on: Mason is probably roughly equivalent to Agents.jl in terms of performance in this case, but our current benchmarking doesn't show this since it calls the program from the shell with the java command.

Unfortunately, I'm not sure how to resolve the issue since I didn't find in the manual of Mason how to keep executing the same model with different seeds and I don't know much about Java itself. cc @Datseris

[Tortar]

Will try again when I find the time, I think with enough effort I could resolve this :-)

(Notice that this doesn't happen with NetLogo and Mesa and indeed the results are more stable)

[Tortar]

Actually I know how we could resolve this:

we can actually use a similar trick I used for the NetLogo benchmarks: use the parameter sweeping facility of Mason, use some java benchmarking (easy one should be enough since we are at least in the order of millisecond which are easy to benchmark) print to file the results and then parse the file.

[Datseris]

how did mason benchmark in the other paper you found out initially before you came into contact with us? maybe we can just use their way.

[Tortar]

I wouldn't follow them: they are using bash for all the packages, this is for Mason itself: https://github.com/isislab-unisa/ABM_Comparison/blob/main/MASON/test.sh . In practice for small benchmarks they are calculating which package is faster to startup not which one is faster to execute the simulation

[Datseris]

well the Mason team knows of the existence of this Repo, you might as well ask them to contribute here by solving this exact issue of not including the startup time.

[Tortar]

Yes I think I will do this for two reasons:

• I manage to use the sweeping facility (hackying it a bit since it requires a dependent variable which I actually don't care about, but I gave a random parameter) but it seems at least as slow as the usage with the java command from the shell. The file I created to do so contains this:

### Parameter Test Example
model = Schelling_small
## Variables
independent = GridWidth
min = 40
max = 40
divisions = 1
dependent = GridWidth
## Sweep Parameters
trials = 100
threads = 1
steps = 20
mod = 0
seed = 42
compress = false
out = /tmp/out.csv

and the average time for simulation is 164.31 ms. It seems equivalent to the one with java.

• I didn't manage to retrieve the total time from the inside of the simulation, changing the main method to:

    public static void main(final String[] args) {
        long startTime = System.nanoTime();
        doLoop((Class)Schelling_large.class, args);
        long endTime   = System.nanoTime();
        long totalTime = endTime - startTime;
        System.out.println(totalTime);
        System.exit(0);
    }

doesn't print anything.

I think the second problem is surely due to inexperience, but it is strange to me that the simulation run is slow even when doing it with the parameter sweeping, so for now I reverted the changes to the timings in the ReadMe since I'm not sure how to address these problems.

Notifying Mason developers so that if they have a fix we can include it: @SeanLuke @spagnuolocarmine @eclab