Potential reduction in performance in systems and problems creations due to new Connector

[gabrevaya]

Benchmarks run on the pm server when not being used by any other demanding task.

Benchmarking code

using Neuroblox
using StochasticDiffEq
using Random
using BenchmarkTools

function benchmarking_generate_system_and_problem(size)
    @info "Size = $size"
    Random.seed!(123)
    N_MSN = round(Int64, 100*size)
    N_FSI = round(Int64, 50*size)
    N_GPe = round(Int64, 80*size)
    N_STN = round(Int64, 40*size)

    global_ns = :g
    @named msn = Striatum_MSN_Adam(namespace=global_ns, N_inhib=N_MSN)
    @named fsi = Striatum_FSI_Adam(namespace=global_ns, N_inhib=N_FSI)
    @named gpe = GPe_Adam(namespace=global_ns, N_inhib=N_GPe)
    @named stn = STN_Adam(namespace=global_ns, N_exci=N_STN)

    ḡ_FSI_MSN = 0.6
    density_FSI_MSN = 0.15
    weight_FSI_MSN = ḡ_FSI_MSN / (N_FSI * density_FSI_MSN)

    ḡ_MSN_GPe = 2.5
    density_MSN_GPe = 0.33
    weight_MSN_GPe = ḡ_MSN_GPe / (N_MSN * density_MSN_GPe)

    ḡ_GPe_STN = 0.3
    density_GPe_STN = 0.05
    weight_GPe_STN = ḡ_GPe_STN / (N_GPe * density_GPe_STN)

    ḡ_STN_FSI = 0.165
    density_STN_FSI = 0.1
    weight_STN_FSI = ḡ_STN_FSI / (N_STN * density_STN_FSI)

    g = MetaDiGraph()
    add_edge!(g, fsi => msn, weight=weight_FSI_MSN, density=density_FSI_MSN)
    add_edge!(g, msn => gpe, weight=weight_MSN_GPe, density=density_MSN_GPe)
    add_edge!(g, gpe => stn, weight=weight_GPe_STN, density=density_GPe_STN)
    add_edge!(g, stn => fsi, weight=weight_STN_FSI, density=density_STN_FSI)

    @info "Benchmarking system_from_graph"
    b1 = @benchmark @named sys = system_from_graph($g);
    display(b1)

    @info "Creating system without benchmarking"
    @named sys = system_from_graph(g);
    tspan = (0.0, 2000.0)
    @info "Benchmarking SDEProblem"
    b2 = @benchmark prob = SDEProblem($sys, [], $tspan, [])
    display(b2)

    return b1, b2
end

b1, b2 = benchmarking_generate_system_and_problem(0.5)

Results with the previous BloxConnector structure (commit 65152b3)

[ Info: Size = 0.1
[ Info: Benchmarking system_from_graph
BenchmarkTools.Trial: 5 samples with 1 evaluation.
 Range (min … max):  1.153 s …   1.225 s  ┊ GC (min … max): 10.02% … 10.26%
 Time  (median):     1.210 s              ┊ GC (median):    10.13%
 Time  (mean ± σ):   1.201 s ± 28.091 ms  ┊ GC (mean ± σ):  10.17% ±  0.15%
  █                                        █   █  █       █  
  █▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁█▁▁▁█▁▁█▁▁▁▁▁▁▁█ ▁
  1.15 s         Histogram: frequency by time        1.22 s <
 Memory estimate: 296.59 MiB, allocs estimate: 5085583.
[ Info: Creating system without benchmarking
[ Info: Benchmarking SDEProblem
BenchmarkTools.Trial: 5 samples with 1 evaluation.
 Range (min … max):  693.776 ms …    1.213 s  ┊ GC (min … max): 10.43% … 49.39%
 Time  (median):        1.150 s               ┊ GC (median):    46.49%
 Time  (mean ± σ):      1.063 s ± 210.003 ms  ┊ GC (mean ± σ):  41.76% ± 16.22%
  █                                              █     ██     █  
  █▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁█▁▁▁▁▁██▁▁▁▁▁█ ▁
  694 ms           Histogram: frequency by time          1.21 s <
 Memory estimate: 459.41 MiB, allocs estimate: 3387199.

[ Info: Size = 0.5
[ Info: Benchmarking system_from_graph
BenchmarkTools.Trial: 1 sample with 1 evaluation.
 Single result which took 15.684 s (10.15% GC) to evaluate,
 with a memory estimate of 4.84 GiB, over 73676707 allocations.
[ Info: Creating system without benchmarking
[ Info: Benchmarking SDEProblem
BenchmarkTools.Trial: 1 sample with 1 evaluation.
 Single result which took 57.951 s (43.41% GC) to evaluate,
 with a memory estimate of 31.74 GiB, over 161161810 allocations.

[ Info: Size = 1.0
[ Info: Benchmarking system_from_graph
BenchmarkTools.Trial: 1 sample with 1 evaluation.
 Single result which took 134.033 s (7.23% GC) to evaluate,
 with a memory estimate of 24.83 GiB, over 360695906 allocations.
[ Info: Creating system without benchmarking
[ Info: Benchmarking SDEProblem
BenchmarkTools.Trial: 1 sample with 1 evaluation.
 Single result which took 495.953 s (32.95% GC) to evaluate,
 with a memory estimate of 242.96 GiB, over 1296997823 allocations.

Results with new Connector structure (commit 9ca1f7e)

[ Info: Size = 0.1
[ Info: Benchmarking system_from_graph
BenchmarkTools.Trial: 4 samples with 1 evaluation.
 Range (min … max):  1.357 s …   1.531 s  ┊ GC (min … max): 3.90% … 8.15%
 Time  (median):     1.506 s              ┊ GC (median):    6.72%
 Time  (mean ± σ):   1.475 s ± 80.236 ms  ┊ GC (mean ± σ):  6.51% ± 2.30%
  █                                            █      █   █  
  █▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁█▁▁▁▁▁▁█▁▁▁█ ▁
  1.36 s         Histogram: frequency by time        1.53 s <
 Memory estimate: 372.34 MiB, allocs estimate: 6388032.
[ Info: Creating system without benchmarking
[ Info: Benchmarking SDEProblem
BenchmarkTools.Trial: 5 samples with 1 evaluation.
 Range (min … max):  889.184 ms …    1.230 s  ┊ GC (min … max):  8.92% … 32.45%
 Time  (median):        1.191 s               ┊ GC (median):    29.86%
 Time  (mean ± σ):      1.125 s ± 143.556 ms  ┊ GC (mean ± σ):  26.09% ±  9.98%
  ▁                                  ▁                  ▁     █  
  █▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁█▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁█▁▁▁▁▁█ ▁
  889 ms           Histogram: frequency by time          1.23 s <
 Memory estimate: 577.62 MiB, allocs estimate: 4587005.

[ Info: Size = 0.5
[ Info: Benchmarking system_from_graph
BenchmarkTools.Trial: 1 sample with 1 evaluation.
 Single result which took 23.343 s (8.45% GC) to evaluate,
 with a memory estimate of 7.13 GiB, over 111617257 allocations.
[ Info: Creating system without benchmarking
[ Info: Benchmarking SDEProblem
BenchmarkTools.Trial: 1 sample with 1 evaluation.
 Single result which took 85.253 s (37.64% GC) to evaluate,
 with a memory estimate of 45.08 GiB, over 273474560 allocations.

[ Info: Size = 1.0
[ Info: Benchmarking system_from_graph
BenchmarkTools.Trial: 1 sample with 1 evaluation.
 Single result which took 154.798 s (9.22% GC) to evaluate,
 with a memory estimate of 43.65 GiB, over 643632448 allocations.
[ Info: Creating system without benchmarking
[ Info: Benchmarking SDEProblem
BenchmarkTools.Trial: 1 sample with 1 evaluation.
 Single result which took 984.163 s (29.73% GC) to evaluate,
 with a memory estimate of 514.59 GiB, over 2684381812 allocations.

Results with new Connector structure (commit 355d24d)

[ Info: Size = 0.1
[ Info: Benchmarking system_from_graph
BenchmarkTools.Trial: 3 samples with 1 evaluation.
 Range (min … max):  2.363 s …   2.466 s  ┊ GC (min … max): 2.54% … 3.10%
 Time  (median):     2.453 s              ┊ GC (median):    3.12%
 Time  (mean ± σ):   2.427 s ± 56.140 ms  ┊ GC (mean ± σ):  3.01% ± 0.42%

  █                                                █      █  
  █▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁█▁▁▁▁▁▁█ ▁
  2.36 s         Histogram: frequency by time        2.47 s <

 Memory estimate: 356.66 MiB, allocs estimate: 6146608.
[ Info: Creating system without benchmarking
[ Info: Benchmarking SDEProblem
BenchmarkTools.Trial: 5 samples with 1 evaluation.
 Range (min … max):  849.435 ms …    1.114 s  ┊ GC (min … max):  7.28% … 27.69%
 Time  (median):        1.104 s               ┊ GC (median):    27.77%
 Time  (mean ± σ):      1.032 s ± 114.695 ms  ┊ GC (mean ± σ):  22.43% ±  9.16%

  ▁                              ▁                          █ ▁  
  █▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁█▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁█▁█ ▁
  849 ms           Histogram: frequency by time          1.11 s <

 Memory estimate: 554.88 MiB, allocs estimate: 4351920.

[ Info: Size = 0.5
[ Info: Benchmarking system_from_graph
BenchmarkTools.Trial: 1 sample with 1 evaluation.
 Single result which took 23.395 s (7.90% GC) to evaluate,
 with a memory estimate of 6.82 GiB, over 107320378 allocations.
[ Info: Creating system without benchmarking                                                                                
[ Info: Benchmarking SDEProblem
BenchmarkTools.Trial: 1 sample with 1 evaluation.
 Single result which took 82.331 s (35.69% GC) to evaluate,
 with a memory estimate of 45.07 GiB, over 273474381 allocations.

[ Info: Size = 1.0
[ Info: Benchmarking system_from_graph
BenchmarkTools.Trial: 1 sample with 1 evaluation.
 Single result which took 142.902 s (9.31% GC) to evaluate,
 with a memory estimate of 40.91 GiB, over 605986071 allocations.
[ Info: Creating system without benchmarking
[ Info: Benchmarking SDEProblem
BenchmarkTools.Trial: 1 sample with 1 evaluation.
 Single result which took 1061.311 s (31.64% GC) to evaluate,
 with a memory estimate of 514.59 GiB, over 2684223173 allocations.

There is also a strange thing happening: when measuring it with @time instead of @Btime or @benchmark from BenchmarkTools, I always get much lower times for the new Connector structure. Below you can find the results from running this

Benchmarking code using @time

using Neuroblox
using StochasticDiffEq
using Random
using BenchmarkTools

function benchmarking_generate_system_and_problem(size)
    @info "Size = $size"
    Random.seed!(123)
    N_MSN = round(Int64, 100*size)
    N_FSI = round(Int64, 50*size)
    N_GPe = round(Int64, 80*size)
    N_STN = round(Int64, 40*size)

    global_ns = :g
    @named msn = Striatum_MSN_Adam(namespace=global_ns, N_inhib=N_MSN)
    @named fsi = Striatum_FSI_Adam(namespace=global_ns, N_inhib=N_FSI)
    @named gpe = GPe_Adam(namespace=global_ns, N_inhib=N_GPe)
    @named stn = STN_Adam(namespace=global_ns, N_exci=N_STN)

    ḡ_FSI_MSN = 0.6
    density_FSI_MSN = 0.15
    weight_FSI_MSN = ḡ_FSI_MSN / (N_FSI * density_FSI_MSN)

    ḡ_MSN_GPe = 2.5
    density_MSN_GPe = 0.33
    weight_MSN_GPe = ḡ_MSN_GPe / (N_MSN * density_MSN_GPe)

    ḡ_GPe_STN = 0.3
    density_GPe_STN = 0.05
    weight_GPe_STN = ḡ_GPe_STN / (N_GPe * density_GPe_STN)

    ḡ_STN_FSI = 0.165
    density_STN_FSI = 0.1
    weight_STN_FSI = ḡ_STN_FSI / (N_STN * density_STN_FSI)

    g = MetaDiGraph()
    add_edge!(g, fsi => msn, weight=weight_FSI_MSN, density=density_FSI_MSN)
    add_edge!(g, msn => gpe, weight=weight_MSN_GPe, density=density_MSN_GPe)
    add_edge!(g, gpe => stn, weight=weight_GPe_STN, density=density_GPe_STN)
    add_edge!(g, stn => fsi, weight=weight_STN_FSI, density=density_STN_FSI)

    @info "Benchmarking system_from_graph"
    @time @named sys = system_from_graph(g);
    tspan = (0.0, 2000.0)
    @info "Benchmarking SDEProblem"
    @time prob = SDEProblem(sys, [], tspan, [])

    return nothing
end

benchmarking_generate_system_and_problem(1.0)

a second time (for avoiding compilation time).

Results with the previous BloxConnector structure using @time (commit 65152b3)

[ Info: Size = 0.1
[ Info: Benchmarking system_from_graph
  0.963136 seconds (5.10 M allocations: 297.931 MiB, 1.64% compilation time)
[ Info: Benchmarking SDEProblem
  1.361199 seconds (3.69 M allocations: 472.257 MiB, 34.37% gc time, 16.43% compilation time)

[ Info: Size = 0.5
[ Info: Benchmarking system_from_graph
 16.066984 seconds (73.79 M allocations: 4.848 GiB, 11.55% gc time, 1.89% compilation time)
[ Info: Benchmarking SDEProblem
 55.454808 seconds (163.23 M allocations: 23.977 GiB, 42.39% gc time, 0.31% compilation time)

[ Info: Size = 1.0
[ Info: Benchmarking system_from_graph
 82.421714 seconds (360.77 M allocations: 24.832 GiB, 11.18% gc time, 0.00% compilation time)
[ Info: Benchmarking SDEProblem
515.491585 seconds (1.31 G allocations: 243.235 GiB, 35.16% gc time, 0.03% compilation time)

Results with new Connector structure using @time (commit 9ca1f7e)

[ Info: Size = 0.1
[ Info: Benchmarking system_from_graph
  1.221776 seconds (5.47 M allocations: 317.271 MiB, 5.78% gc time, 7.77% compilation time)
[ Info: Benchmarking SDEProblem
  1.202553 seconds (3.69 M allocations: 472.449 MiB, 23.38% gc time, 19.28% compilation time)

[ Info: Size = 0.5
[ Info: Benchmarking system_from_graph
 17.208377 seconds (76.42 M allocations: 4.973 GiB, 12.18% gc time, 0.00% compilation time)
[ Info: Benchmarking SDEProblem
 51.666326 seconds (163.11 M allocations: 23.972 GiB, 37.24% gc time, 0.18% compilation time)

[ Info: Size = 1.0
[ Info: Benchmarking system_from_graph
 89.442817 seconds (368.91 M allocations: 25.231 GiB, 10.40% gc time, 0.00% compilation time)
[ Info: Benchmarking SDEProblem
411.526744 seconds (1.30 G allocations: 159.297 GiB, 33.19% gc time)

Results with new Connector structure using @time (commit 355d24d)

[ Info: Size = 0.1
[ Info: Benchmarking system_from_graph
  1.137275 seconds (5.35 M allocations: 310.877 MiB, 6.40% gc time, 0.00% compilation time)
[ Info: Benchmarking SDEProblem
  1.235492 seconds (3.52 M allocations: 463.827 MiB, 43.22% gc time)

[ Info: Size = 0.5
[ Info: Benchmarking system_from_graph
 17.206221 seconds (74.81 M allocations: 4.856 GiB, 11.50% gc time, 0.00% compilation time)
[ Info: Benchmarking SDEProblem
 54.247477 seconds (163.04 M allocations: 23.968 GiB, 37.78% gc time)

[ Info: Size = 1.0
[ Info: Benchmarking system_from_graph
 86.973974 seconds (359.29 M allocations: 24.513 GiB, 9.17% gc time, 4.50% compilation time)
[ Info: Benchmarking SDEProblem
416.883396 seconds (1.30 G allocations: 159.305 GiB, 25.17% gc time, 0.05% compilation time)

Summary for size 1.0:

Version	Function	BenchmarkTools (s)	@time (s)
Previous BloxConnector (65152b3)	system_from_graph	134.033	82.421
Previous BloxConnector (65152b3)	SDEProblem	495.953	515.491
New Connector (9ca1f7e)	system_from_graph	154.798	89.442
New Connector (9ca1f7e)	SDEProblem	984.163	411.526
New Connector (355d24d)	system_from_graph	142.902	86.973
New Connector (355d24d)	SDEProblem	1061.311	416.883