Skip to content

Benchmarks UCX DGX2

Benjamin Zaitlen edited this page Dec 19, 2019 · 4 revisions

GPUS-{1}

Merge benchmark
--------------------------
Chunk-size  | 100000000
Frac-match  | 0.3
Ignore-size | 1.05 MB
Protocol    | ucx
Device(s)   | 1
rmm-pool    | True
tcp         | True
ib          | False
nvlink      | True
==========================
Total time  | 150.19 ms
Total time  | 170.18 ms
Total time  | 151.67 ms
==========================

Worker-Worker Transfer Rates
(w1,w2)     | 25% 50% 75% (total nbytes)
--------------------------

GPUS-{1,2}

Merge benchmark
--------------------------
Chunk-size  | 100000000
Frac-match  | 0.3
Ignore-size | 1.05 MB
Protocol    | ucx
Device(s)   | 1,2
rmm-pool    | True
tcp         | True
ib          | False
nvlink      | True
==========================
Total time  | 1.34 s
Total time  | 1.49 s
Total time  | 1.26 s
==========================

Worker-Worker Transfer Rates
(w1,w2)     | 25% 50% 75% (total nbytes)
--------------------------
(01,02)     | 10.82 GB/s 12.92 GB/s 14.47 GB/s (16.80 GB)
(02,01)     | 12.00 GB/s 13.86 GB/s 16.71 GB/s (16.80 GB)

GPUS-{1,2,3,4}

Merge benchmark
--------------------------
Chunk-size  | 100000000
Frac-match  | 0.3
Ignore-size | 1.05 MB
Protocol    | ucx
Device(s)   | 1,2,3,4
rmm-pool    | True
tcp         | True
ib          | False
nvlink      | True
==========================
Total time  | 2.81 s
Total time  | 2.26 s
Total time  | 2.05 s
==========================

Worker-Worker Transfer Rates
(w1,w2)     | 25% 50% 75% (total nbytes)
--------------------------
(01,02)     | 1.66 GB/s 2.20 GB/s 2.57 GB/s (9.60 GB)
(01,03)     | 1.72 GB/s 2.18 GB/s 5.02 GB/s (11.30 GB)
(01,04)     | 1.55 GB/s 2.18 GB/s 2.68 GB/s (7.20 GB)
(02,01)     | 1.49 GB/s 1.61 GB/s 2.06 GB/s (7.20 GB)
(02,03)     | 1.51 GB/s 1.95 GB/s 2.13 GB/s (9.60 GB)
(02,04)     | 1.54 GB/s 1.84 GB/s 3.94 GB/s (12.00 GB)
(03,01)     | 1.64 GB/s 2.22 GB/s 3.32 GB/s (12.00 GB)
(03,02)     | 1.47 GB/s 1.74 GB/s 2.19 GB/s (7.20 GB)
(03,04)     | 1.56 GB/s 2.07 GB/s 2.64 GB/s (9.60 GB)
(04,01)     | 1.42 GB/s 2.16 GB/s 2.88 GB/s (8.60 GB)
(04,02)     | 1.36 GB/s 2.01 GB/s 2.77 GB/s (12.00 GB)
(04,03)     | 1.55 GB/s 2.10 GB/s 2.64 GB/s (7.90 GB)

GPUS-{0,1,2,3,4,5,6,7}

Merge benchmark
--------------------------
Chunk-size  | 100000000
Frac-match  | 0.3
Ignore-size | 1.05 MB
Protocol    | ucx
Device(s)   | 0,1,2,3,4,5,6,7
rmm-pool    | True
tcp         | True
ib          | False
nvlink      | True
==========================
Total time  | 7.19 s
Total time  | 6.75 s
Total time  | 7.68 s
==========================

Worker-Worker Transfer Rates
(w1,w2)     | 25% 50% 75% (total nbytes)
--------------------------
(00,01)     | 178.84 MB/s 227.69 MB/s 377.47 MB/s (3.60 GB)
(00,02)     | 179.54 MB/s 245.76 MB/s 409.72 MB/s (6.00 GB)
(00,03)     | 172.51 MB/s 209.72 MB/s 365.81 MB/s (3.60 GB)
(00,04)     | 177.00 MB/s 197.57 MB/s 226.20 MB/s (3.60 GB)
(00,05)     | 186.40 MB/s 232.68 MB/s 305.66 MB/s (6.00 GB)
(00,06)     | 178.62 MB/s 204.11 MB/s 316.12 MB/s (3.60 GB)
(00,07)     | 152.06 MB/s 232.53 MB/s 330.15 MB/s (3.60 GB)
(01,00)     | 200.19 MB/s 211.08 MB/s 309.16 MB/s (3.60 GB)
(01,02)     | 195.68 MB/s 215.89 MB/s 272.83 MB/s (3.95 GB)
(01,03)     | 200.94 MB/s 267.06 MB/s 732.91 MB/s (8.40 GB)
(01,04)     | 191.25 MB/s 227.68 MB/s 298.53 MB/s (3.60 GB)
(01,05)     | 182.36 MB/s 233.94 MB/s 318.13 MB/s (3.95 GB)
(01,06)     | 195.38 MB/s 264.65 MB/s 554.62 MB/s (8.40 GB)
(01,07)     | 203.79 MB/s 250.93 MB/s 318.06 MB/s (3.95 GB)
(02,00)     | 199.08 MB/s 223.83 MB/s 599.22 MB/s (3.95 GB)
(02,01)     | 185.86 MB/s 343.45 MB/s 402.57 MB/s (3.60 GB)
(02,03)     | 193.91 MB/s 266.90 MB/s 545.83 MB/s (3.95 GB)
(02,04)     | 172.06 MB/s 346.57 MB/s 668.63 MB/s (6.35 GB)
(02,05)     | 168.25 MB/s 296.38 MB/s 394.16 MB/s (3.95 GB)
(02,06)     | 199.81 MB/s 258.27 MB/s 388.68 MB/s (3.60 GB)
(02,07)     | 193.64 MB/s 343.76 MB/s 363.90 MB/s (3.60 GB)
(03,00)     | 169.60 MB/s 303.95 MB/s 657.70 MB/s (6.35 GB)
(03,01)     | 161.35 MB/s 328.87 MB/s 507.69 MB/s (3.95 GB)
(03,02)     | 174.39 MB/s 347.08 MB/s 733.39 MB/s (6.35 GB)
(03,04)     | 167.69 MB/s 310.12 MB/s 486.60 MB/s (6.35 GB)
(03,05)     | 156.70 MB/s 300.17 MB/s 450.68 MB/s (3.95 GB)
(03,06)     | 171.48 MB/s 275.04 MB/s 407.68 MB/s (3.95 GB)
(03,07)     | 161.11 MB/s 298.84 MB/s 441.46 MB/s (3.95 GB)
(04,00)     | 206.85 MB/s 309.76 MB/s 506.37 MB/s (3.60 GB)
(04,01)     | 152.33 MB/s 288.52 MB/s 387.89 MB/s (3.95 GB)
(04,02)     | 209.23 MB/s 580.82 MB/s 683.98 MB/s (6.00 GB)
(04,03)     | 169.44 MB/s 278.97 MB/s 560.80 MB/s (3.95 GB)
(04,05)     | 177.65 MB/s 266.17 MB/s 279.76 MB/s (3.60 GB)
(04,06)     | 181.01 MB/s 291.46 MB/s 505.72 MB/s (6.35 GB)
(04,07)     | 266.53 MB/s 535.57 MB/s 676.31 MB/s (4.30 GB)
(05,00)     | 164.41 MB/s 280.95 MB/s 647.22 MB/s (3.95 GB)
(05,01)     | 173.94 MB/s 431.95 MB/s 716.22 MB/s (6.35 GB)
(05,02)     | 176.26 MB/s 232.37 MB/s 375.65 MB/s (3.60 GB)
(05,03)     | 195.91 MB/s 287.24 MB/s 430.93 MB/s (6.00 GB)
(05,04)     | 155.28 MB/s 326.41 MB/s 456.20 MB/s (3.60 GB)
(05,06)     | 140.38 MB/s 371.68 MB/s 391.61 MB/s (3.60 GB)
(05,07)     | 164.48 MB/s 350.46 MB/s 440.96 MB/s (6.35 GB)
(06,00)     | 202.99 MB/s 255.52 MB/s 585.96 MB/s (4.65 GB)
(06,01)     | 217.55 MB/s 399.81 MB/s 885.23 MB/s (5.00 GB)
(06,02)     | 205.47 MB/s 258.67 MB/s 690.34 MB/s (4.30 GB)
(06,03)     | 200.95 MB/s 275.62 MB/s 931.80 MB/s (7.05 GB)
(06,04)     | 182.58 MB/s 311.83 MB/s 503.52 MB/s (4.30 GB)
(06,05)     | 177.93 MB/s 333.34 MB/s 892.19 MB/s (4.65 GB)
(06,07)     | 203.24 MB/s 264.84 MB/s 895.85 MB/s (4.65 GB)
(07,00)     | 206.70 MB/s 263.56 MB/s 568.15 MB/s (6.35 GB)
(07,01)     | 230.59 MB/s 255.88 MB/s 397.75 MB/s (3.95 GB)
(07,02)     | 185.74 MB/s 244.85 MB/s 341.24 MB/s (3.60 GB)
(07,03)     | 215.62 MB/s 220.91 MB/s 342.90 MB/s (3.60 GB)
(07,04)     | 178.25 MB/s 216.32 MB/s 380.37 MB/s (3.25 GB)
(07,05)     | 207.89 MB/s 283.95 MB/s 487.75 MB/s (8.75 GB)
(07,06)     | 188.80 MB/s 251.82 MB/s 335.39 MB/s (3.95 GB)

GPUS-{0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15}

Merge benchmark
--------------------------
Chunk-size  | 1000000
Frac-match  | 0.3
Ignore-size | 1.05 MB
Protocol    | ucx
Device(s)   | 0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15
rmm-pool    | True
tcp         | True
ib          | False
nvlink      | True
==========================
Total time  | 14.25 s
Total time  | 15.64 s
Total time  | 27.38 s
==========================

Worker-Worker Transfer Rates
(w1,w2)     | 25% 50% 75% (total nbytes)
--------------------------
(00,01)     | 402.74 kB/s 570.72 kB/s 1.75 MB/s (18.01 MB)
(00,02)     | 378.21 kB/s 494.22 kB/s 1.22 MB/s (15.48 MB)
(00,03)     | 426.70 kB/s 496.69 kB/s 1.52 MB/s (16.27 MB)
(00,04)     | 426.87 kB/s 501.78 kB/s 1.51 MB/s (18.03 MB)
(00,05)     | 498.77 kB/s 579.11 kB/s 1.73 MB/s (16.26 MB)
(00,06)     | 426.72 kB/s 527.25 kB/s 2.04 MB/s (36.48 MB)
(00,07)     | 431.42 kB/s 509.97 kB/s 1.27 MB/s (11.97 MB)
(00,08)     | 481.40 kB/s 639.79 kB/s 1.31 MB/s (20.51 MB)
(00,09)     | 426.18 kB/s 496.75 kB/s 762.24 kB/s (18.76 MB)
(00,10)     | 485.24 kB/s 522.06 kB/s 1.53 MB/s (42.79 MB)
(00,11)     | 390.25 kB/s 502.77 kB/s 1.02 MB/s (14.48 MB)
(00,12)     | 374.52 kB/s 530.01 kB/s 1.66 MB/s (38.50 MB)
(00,13)     | 425.80 kB/s 512.62 kB/s 1.74 MB/s (16.25 MB)
(00,14)     | 485.53 kB/s 590.16 kB/s 1.52 MB/s (40.74 MB)
(00,15)     | 459.09 kB/s 486.30 kB/s 999.41 kB/s (14.50 MB)
(01,00)     | 415.85 kB/s 426.02 kB/s 653.87 kB/s (14.50 MB)
(01,02)     | 368.25 kB/s 498.30 kB/s 643.53 kB/s (13.77 MB)
(01,03)     | 422.11 kB/s 661.34 kB/s 912.76 kB/s (16.25 MB)
(01,04)     | 449.34 kB/s 508.04 kB/s 624.64 kB/s (17.98 MB)
(01,05)     | 427.83 kB/s 559.32 kB/s 897.07 kB/s (16.24 MB)
(01,06)     | 388.48 kB/s 587.17 kB/s 1.48 MB/s (58.71 MB)
(01,07)     | 424.50 kB/s 454.18 kB/s 2.15 MB/s (12.00 MB)
(01,08)     | 418.19 kB/s 503.30 kB/s 795.56 kB/s (20.50 MB)
(01,09)     | 489.51 kB/s 565.02 kB/s 781.27 kB/s (18.75 MB)
(01,10)     | 452.20 kB/s 500.39 kB/s 541.70 kB/s (16.99 MB)
(01,11)     | 411.91 kB/s 516.48 kB/s 819.27 kB/s (25.04 MB)
(01,12)     | 340.25 kB/s 448.63 kB/s 732.87 kB/s (12.73 MB)
(01,13)     | 446.02 kB/s 660.65 kB/s 1.28 MB/s (16.25 MB)
(01,14)     | 422.56 kB/s 600.38 kB/s 813.02 kB/s (40.75 MB)
(01,15)     | 435.73 kB/s 516.57 kB/s 1.20 MB/s (28.74 MB)
(02,00)     | 394.76 kB/s 539.50 kB/s 779.27 kB/s (13.72 MB)
(02,01)     | 449.40 kB/s 649.08 kB/s 1.00 MB/s (38.24 MB)
(02,03)     | 409.51 kB/s 497.02 kB/s 792.21 kB/s (15.49 MB)
(02,04)     | 422.52 kB/s 554.02 kB/s 729.16 kB/s (17.26 MB)
(02,05)     | 373.96 kB/s 473.78 kB/s 738.10 kB/s (15.49 MB)
(02,06)     | 448.56 kB/s 543.20 kB/s 869.10 kB/s (17.24 MB)
(02,07)     | 396.16 kB/s 642.57 kB/s 713.69 kB/s (13.71 MB)
(02,08)     | 384.85 kB/s 640.15 kB/s 1.11 MB/s (22.26 MB)
(02,09)     | 401.26 kB/s 648.04 kB/s 1.81 MB/s (69.45 MB)
(02,10)     | 388.11 kB/s 543.10 kB/s 889.04 kB/s (20.51 MB)
(02,11)     | 431.68 kB/s 679.30 kB/s 2.09 MB/s (52.49 MB)
(02,12)     | 408.94 kB/s 541.91 kB/s 804.21 kB/s (13.72 MB)
(02,13)     | 429.45 kB/s 546.12 kB/s 1.25 MB/s (17.25 MB)
(02,14)     | 424.11 kB/s 498.90 kB/s 731.90 kB/s (17.25 MB)
(02,15)     | 425.17 kB/s 597.96 kB/s 1.18 MB/s (37.72 MB)
(03,00)     | 456.09 kB/s 838.20 kB/s 990.81 kB/s (37.27 MB)
(03,01)     | 454.59 kB/s 580.18 kB/s 988.72 kB/s (19.75 MB)
(03,02)     | 537.79 kB/s 739.69 kB/s 1.35 MB/s (43.01 MB)
(03,04)     | 450.75 kB/s 574.64 kB/s 1.47 MB/s (19.74 MB)
(03,05)     | 454.75 kB/s 525.40 kB/s 845.52 kB/s (18.01 MB)
(03,06)     | 405.18 kB/s 716.87 kB/s 1.37 MB/s (39.48 MB)
(03,07)     | 454.25 kB/s 463.23 kB/s 780.76 kB/s (13.76 MB)
(03,08)     | 455.80 kB/s 548.61 kB/s 690.46 kB/s (25.73 MB)
(03,09)     | 456.49 kB/s 585.34 kB/s 918.86 kB/s (65.61 MB)
(03,10)     | 469.39 kB/s 668.70 kB/s 1.30 MB/s (44.48 MB)
(03,11)     | 342.28 kB/s 689.96 kB/s 918.76 kB/s (16.28 MB)
(03,12)     | 392.36 kB/s 751.80 kB/s 989.50 kB/s (40.74 MB)
(03,13)     | 454.16 kB/s 623.10 kB/s 1.06 MB/s (19.73 MB)
(03,14)     | 450.04 kB/s 561.97 kB/s 967.34 kB/s (19.74 MB)
(03,15)     | 419.21 kB/s 584.38 kB/s 977.00 kB/s (16.25 MB)
(04,00)     | 497.53 kB/s 534.11 kB/s 928.19 kB/s (12.74 MB)
(04,01)     | 506.49 kB/s 555.12 kB/s 799.37 kB/s (40.20 MB)
(04,02)     | 381.68 kB/s 505.31 kB/s 556.60 kB/s (13.76 MB)
(04,03)     | 409.42 kB/s 492.81 kB/s 641.13 kB/s (14.48 MB)
(04,05)     | 421.16 kB/s 496.56 kB/s 656.04 kB/s (14.49 MB)
(04,06)     | 380.09 kB/s 427.16 kB/s 769.33 kB/s (35.98 MB)
(04,07)     | 399.95 kB/s 414.96 kB/s 643.51 kB/s (10.23 MB)
(04,08)     | 357.40 kB/s 411.09 kB/s 558.28 kB/s (15.25 MB)
(04,09)     | 457.00 kB/s 514.68 kB/s 581.44 kB/s (17.01 MB)
(04,10)     | 408.28 kB/s 429.04 kB/s 578.04 kB/s (17.00 MB)
(04,11)     | 423.46 kB/s 504.16 kB/s 785.47 kB/s (12.75 MB)
(04,12)     | 383.21 kB/s 496.03 kB/s 913.86 kB/s (28.76 MB)
(04,13)     | 425.04 kB/s 517.69 kB/s 895.74 kB/s (38.49 MB)
(04,14)     | 449.44 kB/s 513.79 kB/s 721.65 kB/s (16.27 MB)
(04,15)     | 449.60 kB/s 511.98 kB/s 1.00 MB/s (37.28 MB)
(05,00)     | 471.99 kB/s 513.96 kB/s 931.06 kB/s (17.99 MB)
(05,01)     | 500.23 kB/s 560.24 kB/s 843.78 kB/s (21.51 MB)
(05,02)     | 458.23 kB/s 549.54 kB/s 619.75 kB/s (19.04 MB)
(05,03)     | 460.91 kB/s 544.03 kB/s 1.04 MB/s (19.75 MB)
(05,04)     | 460.74 kB/s 528.02 kB/s 1.03 MB/s (21.48 MB)
(05,06)     | 440.18 kB/s 501.58 kB/s 683.06 kB/s (17.26 MB)
(05,07)     | 470.90 kB/s 543.00 kB/s 2.10 MB/s (63.57 MB)
(05,08)     | 437.95 kB/s 500.04 kB/s 696.28 kB/s (20.52 MB)
(05,09)     | 464.13 kB/s 526.20 kB/s 645.25 kB/s (22.24 MB)
(05,10)     | 460.27 kB/s 552.87 kB/s 942.44 kB/s (24.03 MB)
(05,11)     | 458.62 kB/s 553.72 kB/s 932.79 kB/s (18.00 MB)
(05,12)     | 459.99 kB/s 582.77 kB/s 1.18 MB/s (18.01 MB)
(05,13)     | 486.14 kB/s 610.41 kB/s 1.11 MB/s (42.52 MB)
(05,14)     | 469.75 kB/s 568.15 kB/s 766.10 kB/s (21.52 MB)
(05,15)     | 457.19 kB/s 503.66 kB/s 923.12 kB/s (17.99 MB)
(06,00)     | 383.24 kB/s 770.63 kB/s 1.15 MB/s (13.75 MB)
(06,01)     | 407.54 kB/s 806.43 kB/s 1.16 MB/s (17.22 MB)
(06,02)     | 457.82 kB/s 990.41 kB/s 1.18 MB/s (17.27 MB)
(06,03)     | 406.16 kB/s 1.01 MB/s 1.43 MB/s (15.48 MB)
(06,04)     | 361.78 kB/s 1.09 MB/s 1.23 MB/s (18.97 MB)
(06,05)     | 516.40 kB/s 884.66 kB/s 2.05 MB/s (60.48 MB)
(06,07)     | 523.13 kB/s 683.99 kB/s 1.23 MB/s (13.77 MB)
(06,08)     | 379.03 kB/s 1.08 MB/s 1.49 MB/s (13.71 MB)
(06,09)     | 557.05 kB/s 989.40 kB/s 1.88 MB/s (46.23 MB)
(06,10)     | 555.16 kB/s 1.15 MB/s 2.88 MB/s (75.06 MB)
(06,11)     | 394.34 kB/s 1.02 MB/s 1.22 MB/s (13.75 MB)
(06,12)     | 363.61 kB/s 1.15 MB/s 1.28 MB/s (15.49 MB)
(06,13)     | 420.70 kB/s 886.83 kB/s 1.20 MB/s (15.49 MB)
(06,14)     | 425.91 kB/s 1.15 MB/s 1.35 MB/s (41.18 MB)
(06,15)     | 415.64 kB/s 683.52 kB/s 831.73 kB/s (13.76 MB)
(07,00)     | 443.87 kB/s 552.64 kB/s 716.25 kB/s (38.29 MB)
(07,01)     | 471.05 kB/s 585.38 kB/s 721.85 kB/s (17.25 MB)
(07,02)     | 495.48 kB/s 590.71 kB/s 914.31 kB/s (17.26 MB)
(07,03)     | 425.35 kB/s 475.63 kB/s 552.93 kB/s (15.49 MB)
(07,04)     | 492.86 kB/s 630.91 kB/s 721.94 kB/s (17.25 MB)
(07,05)     | 400.21 kB/s 458.53 kB/s 685.97 kB/s (15.50 MB)
(07,06)     | 449.55 kB/s 537.51 kB/s 644.29 kB/s (15.51 MB)
(07,08)     | 387.88 kB/s 472.99 kB/s 639.31 kB/s (42.98 MB)
(07,09)     | 454.12 kB/s 460.64 kB/s 914.93 kB/s (39.54 MB)
(07,10)     | 426.40 kB/s 514.42 kB/s 749.08 kB/s (15.50 MB)
(07,11)     | 490.54 kB/s 617.75 kB/s 836.87 kB/s (13.74 MB)
(07,12)     | 366.47 kB/s 585.50 kB/s 882.06 kB/s (37.74 MB)
(07,13)     | 458.60 kB/s 529.46 kB/s 681.26 kB/s (15.49 MB)
(07,14)     | 430.85 kB/s 532.27 kB/s 643.50 kB/s (17.23 MB)
(07,15)     | 473.45 kB/s 577.95 kB/s 672.22 kB/s (13.76 MB)
(08,00)     | 392.81 kB/s 586.77 kB/s 963.64 kB/s (46.21 MB)
(08,01)     | 402.94 kB/s 578.93 kB/s 984.79 kB/s (49.72 MB)
(08,02)     | 383.88 kB/s 534.06 kB/s 831.94 kB/s (20.74 MB)
(08,03)     | 402.85 kB/s 619.93 kB/s 922.65 kB/s (48.01 MB)
(08,04)     | 446.32 kB/s 562.27 kB/s 883.26 kB/s (25.78 MB)
(08,05)     | 403.35 kB/s 517.20 kB/s 982.81 kB/s (48.01 MB)
(08,06)     | 374.51 kB/s 425.12 kB/s 781.72 kB/s (19.01 MB)
(08,07)     | 371.42 kB/s 500.61 kB/s 816.76 kB/s (15.49 MB)
(08,09)     | 392.89 kB/s 471.01 kB/s 985.20 kB/s (29.00 MB)
(08,10)     | 392.00 kB/s 464.40 kB/s 999.46 kB/s (28.97 MB)
(08,11)     | 395.97 kB/s 618.34 kB/s 987.26 kB/s (48.49 MB)
(08,12)     | 363.14 kB/s 605.21 kB/s 945.18 kB/s (20.48 MB)
(08,13)     | 391.24 kB/s 474.36 kB/s 1.06 MB/s (24.02 MB)
(08,14)     | 391.74 kB/s 548.45 kB/s 925.37 kB/s (25.72 MB)
(08,15)     | 372.19 kB/s 566.57 kB/s 917.36 kB/s (20.47 MB)
(09,00)     | 505.43 kB/s 567.49 kB/s 897.51 kB/s (20.51 MB)
(09,01)     | 463.93 kB/s 607.85 kB/s 847.07 kB/s (24.04 MB)
(09,02)     | 462.95 kB/s 757.42 kB/s 1.46 MB/s (42.98 MB)
(09,03)     | 473.96 kB/s 512.56 kB/s 848.38 kB/s (22.23 MB)
(09,04)     | 474.08 kB/s 582.55 kB/s 849.44 kB/s (24.01 MB)
(09,05)     | 489.11 kB/s 601.66 kB/s 1.07 MB/s (45.07 MB)
(09,06)     | 500.50 kB/s 516.44 kB/s 1.51 MB/s (17.22 MB)
(09,07)     | 495.59 kB/s 504.65 kB/s 1.66 MB/s (39.47 MB)
(09,08)     | 407.73 kB/s 507.05 kB/s 1.33 MB/s (25.77 MB)
(09,10)     | 458.61 kB/s 722.87 kB/s 1.25 MB/s (22.23 MB)
(09,11)     | 419.67 kB/s 622.64 kB/s 1.20 MB/s (44.47 MB)
(09,12)     | 456.59 kB/s 568.23 kB/s 1.04 MB/s (20.52 MB)
(09,13)     | 508.26 kB/s 620.39 kB/s 1.27 MB/s (22.24 MB)
(09,14)     | 505.29 kB/s 515.59 kB/s 800.62 kB/s (24.03 MB)
(09,15)     | 448.49 kB/s 549.97 kB/s 732.86 kB/s (20.51 MB)
(10,00)     | 432.88 kB/s 485.69 kB/s 704.13 kB/s (17.01 MB)
(10,01)     | 451.03 kB/s 532.16 kB/s 815.37 kB/s (44.51 MB)
(10,02)     | 394.73 kB/s 524.00 kB/s 777.32 kB/s (38.22 MB)
(10,03)     | 485.08 kB/s 647.79 kB/s 903.47 kB/s (18.77 MB)
(10,04)     | 439.27 kB/s 571.93 kB/s 829.67 kB/s (20.51 MB)
(10,05)     | 442.67 kB/s 486.57 kB/s 647.15 kB/s (18.75 MB)
(10,06)     | 397.50 kB/s 516.49 kB/s 721.23 kB/s (15.50 MB)
(10,07)     | 445.10 kB/s 455.93 kB/s 562.91 kB/s (12.00 MB)
(10,08)     | 452.87 kB/s 765.62 kB/s 990.11 kB/s (75.69 MB)
(10,09)     | 431.95 kB/s 454.64 kB/s 717.23 kB/s (16.99 MB)
(10,11)     | 412.40 kB/s 622.54 kB/s 731.49 kB/s (16.98 MB)
(10,12)     | 351.16 kB/s 646.61 kB/s 1.45 MB/s (18.76 MB)
(10,13)     | 456.02 kB/s 487.94 kB/s 1.25 MB/s (67.26 MB)
(10,14)     | 439.29 kB/s 559.25 kB/s 645.33 kB/s (20.52 MB)
(10,15)     | 439.29 kB/s 648.45 kB/s 829.26 kB/s (18.75 MB)
(11,00)     | 479.69 kB/s 577.87 kB/s 975.73 kB/s (18.00 MB)
(11,01)     | 403.68 kB/s 621.44 kB/s 973.29 kB/s (19.75 MB)
(11,02)     | 426.26 kB/s 641.58 kB/s 922.94 kB/s (20.72 MB)
(11,03)     | 510.13 kB/s 1.03 MB/s 1.22 MB/s (16.24 MB)
(11,04)     | 511.03 kB/s 650.56 kB/s 1.00 MB/s (18.01 MB)
(11,05)     | 548.98 kB/s 904.90 kB/s 1.05 MB/s (40.23 MB)
(11,06)     | 434.68 kB/s 569.56 kB/s 818.62 kB/s (15.50 MB)
(11,07)     | 503.33 kB/s 581.24 kB/s 705.81 kB/s (13.72 MB)
(11,08)     | 461.22 kB/s 706.21 kB/s 935.99 kB/s (20.51 MB)
(11,09)     | 550.03 kB/s 699.14 kB/s 869.98 kB/s (39.78 MB)
(11,10)     | 402.35 kB/s 674.28 kB/s 856.07 kB/s (20.46 MB)
(11,12)     | 402.67 kB/s 724.28 kB/s 1.04 MB/s (16.26 MB)
(11,13)     | 403.75 kB/s 690.51 kB/s 972.86 kB/s (18.01 MB)
(11,14)     | 477.21 kB/s 681.59 kB/s 1.07 MB/s (62.99 MB)
(11,15)     | 405.34 kB/s 686.77 kB/s 946.60 kB/s (18.00 MB)
(12,00)     | 436.70 kB/s 602.66 kB/s 752.25 kB/s (16.23 MB)
(12,01)     | 417.38 kB/s 686.97 kB/s 1.31 MB/s (40.76 MB)
(12,02)     | 461.60 kB/s 554.70 kB/s 884.15 kB/s (41.26 MB)
(12,03)     | 460.40 kB/s 549.51 kB/s 911.55 kB/s (18.02 MB)
(12,04)     | 413.55 kB/s 642.07 kB/s 956.66 kB/s (44.25 MB)
(12,05)     | 430.51 kB/s 678.04 kB/s 852.49 kB/s (19.76 MB)
(12,06)     | 438.44 kB/s 514.04 kB/s 676.78 kB/s (15.50 MB)
(12,07)     | 378.74 kB/s 549.84 kB/s 771.74 kB/s (13.79 MB)
(12,08)     | 356.72 kB/s 506.67 kB/s 695.85 kB/s (24.02 MB)
(12,09)     | 378.48 kB/s 551.26 kB/s 692.50 kB/s (20.49 MB)
(12,10)     | 466.25 kB/s 544.61 kB/s 715.82 kB/s (22.26 MB)
(12,11)     | 461.48 kB/s 592.82 kB/s 1.38 MB/s (40.24 MB)
(12,13)     | 377.82 kB/s 550.68 kB/s 977.39 kB/s (17.98 MB)
(12,14)     | 379.76 kB/s 593.34 kB/s 705.31 kB/s (19.75 MB)
(12,15)     | 372.90 kB/s 692.62 kB/s 861.69 kB/s (40.22 MB)
(13,00)     | 505.61 kB/s 821.78 kB/s 1.35 MB/s (67.75 MB)
(13,01)     | 512.86 kB/s 795.87 kB/s 1.13 MB/s (23.25 MB)
(13,02)     | 466.95 kB/s 646.26 kB/s 1.15 MB/s (20.72 MB)
(13,03)     | 454.26 kB/s 825.04 kB/s 1.09 MB/s (21.51 MB)
(13,04)     | 500.75 kB/s 666.30 kB/s 968.34 kB/s (23.27 MB)
(13,05)     | 482.51 kB/s 644.84 kB/s 1.12 MB/s (21.49 MB)
(13,06)     | 387.98 kB/s 692.61 kB/s 1.21 MB/s (18.99 MB)
(13,07)     | 456.97 kB/s 775.48 kB/s 1.09 MB/s (17.25 MB)
(13,08)     | 497.15 kB/s 790.64 kB/s 1.40 MB/s (49.75 MB)
(13,09)     | 491.33 kB/s 645.89 kB/s 893.42 kB/s (24.00 MB)
(13,10)     | 465.61 kB/s 639.94 kB/s 1.27 MB/s (69.00 MB)
(13,11)     | 464.73 kB/s 860.18 kB/s 1.20 MB/s (19.75 MB)
(13,12)     | 411.93 kB/s 790.63 kB/s 1.31 MB/s (19.77 MB)
(13,14)     | 500.05 kB/s 695.13 kB/s 1.19 MB/s (23.25 MB)
(13,15)     | 497.49 kB/s 994.02 kB/s 2.11 MB/s (44.22 MB)
(14,00)     | 484.53 kB/s 641.38 kB/s 969.86 kB/s (32.25 MB)
(14,01)     | 444.85 kB/s 547.21 kB/s 816.71 kB/s (19.74 MB)
(14,02)     | 453.25 kB/s 604.92 kB/s 1.79 MB/s (63.96 MB)
(14,03)     | 413.00 kB/s 498.29 kB/s 823.65 kB/s (17.99 MB)
(14,04)     | 484.75 kB/s 556.01 kB/s 824.16 kB/s (19.70 MB)
(14,05)     | 440.82 kB/s 623.16 kB/s 815.63 kB/s (17.98 MB)
(14,06)     | 400.45 kB/s 469.07 kB/s 804.95 kB/s (15.50 MB)
(14,07)     | 407.13 kB/s 623.86 kB/s 1.20 MB/s (37.76 MB)
(14,08)     | 355.53 kB/s 455.34 kB/s 747.87 kB/s (20.48 MB)
(14,09)     | 442.92 kB/s 669.43 kB/s 784.52 kB/s (20.50 MB)
(14,10)     | 443.20 kB/s 680.99 kB/s 736.90 kB/s (20.52 MB)
(14,11)     | 436.52 kB/s 635.26 kB/s 858.89 kB/s (16.26 MB)
(14,12)     | 419.07 kB/s 636.74 kB/s 854.93 kB/s (16.24 MB)
(14,13)     | 418.90 kB/s 644.69 kB/s 1.01 MB/s (42.00 MB)
(14,15)     | 474.14 kB/s 634.02 kB/s 946.25 kB/s (16.23 MB)
(15,00)     | 370.90 kB/s 517.91 kB/s 741.75 kB/s (16.24 MB)
(15,01)     | 494.68 kB/s 879.31 kB/s 980.93 kB/s (43.78 MB)
(15,02)     | 368.73 kB/s 518.47 kB/s 860.71 kB/s (17.23 MB)
(15,03)     | 374.36 kB/s 770.26 kB/s 1.17 MB/s (17.98 MB)
(15,04)     | 420.32 kB/s 794.66 kB/s 970.38 kB/s (42.48 MB)
(15,05)     | 395.68 kB/s 741.97 kB/s 1.06 MB/s (42.01 MB)
(15,06)     | 385.47 kB/s 571.73 kB/s 1.19 MB/s (39.97 MB)
(15,07)     | 481.07 kB/s 572.03 kB/s 992.67 kB/s (13.78 MB)
(15,08)     | 437.55 kB/s 635.48 kB/s 1.05 MB/s (46.21 MB)
(15,09)     | 470.68 kB/s 585.31 kB/s 937.94 kB/s (20.50 MB)
(15,10)     | 440.30 kB/s 676.89 kB/s 1.04 MB/s (44.48 MB)
(15,11)     | 371.78 kB/s 517.99 kB/s 742.15 kB/s (16.24 MB)
(15,12)     | 373.76 kB/s 529.44 kB/s 741.56 kB/s (16.25 MB)
(15,13)     | 385.57 kB/s 604.91 kB/s 1.07 MB/s (17.99 MB)
(15,14)     | 462.26 kB/s 768.21 kB/s 934.75 kB/s (19.75 MB)