Re: CPUs for new databases - Mailing list pgsql-performance

On 2010-10-27 21:58, Greg Smith wrote:
> Ivan Voras wrote:
>> FWIW, yes - once the IO is fast enough or not necessary (e.g. the
>> read-mostly database fits in RAM), RAM bandwidth *is* the next
>> bottleneck and it really, really can be observed in actual loads.
>
> This is exactly what I've concluded, after many rounds of correlating
> memory speed tests with pgbench tests against in-RAM databases.  And
> it's the reason why I've written the stream-scaling utility and been
> collecting test results from as many systems as possible.  That seemed
> to get dismissed upthread as not being the answer the poster was
> looking for, but I think you have to get a handle on that part before
> the rest of the trivia involved even matters.
>
> I have a bunch more results that have been flowing in that I need to
> publish there soon.  Note that there is a bug in stream-scaling where
> sufficiently large systems can hit a compiler problem where it reports
> "relocation truncated to fit: R_X86_64_PC32 against `.bss'".  I have
> two of those reports and am working on resolving.
>

Just to chime in after the new systems were purchased and installed. We
ended up with buying a 4x Opteron 6168 (12core Magny-cours,12MB cache @
1.9Ghz) with 128GB 1333Mhz DDR3 RAM. That's an insane 48 cores. That is
perhaps slightly beyond the scaling horizon for Postgres at the moment
but we're confident that scaling will improve over the lifetime with
these servers.

Using the stream-scaling test we see some very impressive numbers:

Highest results comes at 32 threads:

Number of Threads requested = 32
Function      Rate (MB/s)   Avg time     Min time     Max time
Triad:      81013.5506       0.0378       0.0377       0.0379

The pattern is quite clear in that any multiple of 4 (the number of
physical CPU packages) get a higher value but thinking about how the
memory is connected and utilized this makes perfect sense.

Full output below

Regards,
Christian Elmerot, One.com




=== CPU cache information ===
CPU /sys/devices/system/cpu/cpu0 Level 1 Cache: 64K (Data)
CPU /sys/devices/system/cpu/cpu0 Level 1 Cache: 64K (Instruction)
CPU /sys/devices/system/cpu/cpu0 Level 2 Cache: 512K (Unified)
CPU /sys/devices/system/cpu/cpu0 Level 3 Cache: 5118K (Unified)
CPU /sys/devices/system/cpu/cpu1 Level 1 Cache: 64K (Data)
CPU /sys/devices/system/cpu/cpu1 Level 1 Cache: 64K (Instruction)
CPU /sys/devices/system/cpu/cpu1 Level 2 Cache: 512K (Unified)
CPU /sys/devices/system/cpu/cpu1 Level 3 Cache: 5118K (Unified)
CPU /sys/devices/system/cpu/cpu10 Level 1 Cache: 64K (Data)
CPU /sys/devices/system/cpu/cpu10 Level 1 Cache: 64K (Instruction)
CPU /sys/devices/system/cpu/cpu10 Level 2 Cache: 512K (Unified)
CPU /sys/devices/system/cpu/cpu10 Level 3 Cache: 5118K (Unified)
CPU /sys/devices/system/cpu/cpu11 Level 1 Cache: 64K (Data)
CPU /sys/devices/system/cpu/cpu11 Level 1 Cache: 64K (Instruction)
CPU /sys/devices/system/cpu/cpu11 Level 2 Cache: 512K (Unified)
CPU /sys/devices/system/cpu/cpu11 Level 3 Cache: 5118K (Unified)
CPU /sys/devices/system/cpu/cpu12 Level 1 Cache: 64K (Data)
CPU /sys/devices/system/cpu/cpu12 Level 1 Cache: 64K (Instruction)
CPU /sys/devices/system/cpu/cpu12 Level 2 Cache: 512K (Unified)
CPU /sys/devices/system/cpu/cpu12 Level 3 Cache: 5118K (Unified)
CPU /sys/devices/system/cpu/cpu13 Level 1 Cache: 64K (Data)
CPU /sys/devices/system/cpu/cpu13 Level 1 Cache: 64K (Instruction)
CPU /sys/devices/system/cpu/cpu13 Level 2 Cache: 512K (Unified)
CPU /sys/devices/system/cpu/cpu13 Level 3 Cache: 5118K (Unified)
CPU /sys/devices/system/cpu/cpu14 Level 1 Cache: 64K (Data)
CPU /sys/devices/system/cpu/cpu14 Level 1 Cache: 64K (Instruction)
CPU /sys/devices/system/cpu/cpu14 Level 2 Cache: 512K (Unified)
CPU /sys/devices/system/cpu/cpu14 Level 3 Cache: 5118K (Unified)
CPU /sys/devices/system/cpu/cpu15 Level 1 Cache: 64K (Data)
CPU /sys/devices/system/cpu/cpu15 Level 1 Cache: 64K (Instruction)
CPU /sys/devices/system/cpu/cpu15 Level 2 Cache: 512K (Unified)
CPU /sys/devices/system/cpu/cpu15 Level 3 Cache: 5118K (Unified)
CPU /sys/devices/system/cpu/cpu16 Level 1 Cache: 64K (Data)
CPU /sys/devices/system/cpu/cpu16 Level 1 Cache: 64K (Instruction)
CPU /sys/devices/system/cpu/cpu16 Level 2 Cache: 512K (Unified)
CPU /sys/devices/system/cpu/cpu16 Level 3 Cache: 5118K (Unified)
CPU /sys/devices/system/cpu/cpu17 Level 1 Cache: 64K (Data)
CPU /sys/devices/system/cpu/cpu17 Level 1 Cache: 64K (Instruction)
CPU /sys/devices/system/cpu/cpu17 Level 2 Cache: 512K (Unified)
CPU /sys/devices/system/cpu/cpu17 Level 3 Cache: 5118K (Unified)
CPU /sys/devices/system/cpu/cpu18 Level 1 Cache: 64K (Data)
CPU /sys/devices/system/cpu/cpu18 Level 1 Cache: 64K (Instruction)
CPU /sys/devices/system/cpu/cpu18 Level 2 Cache: 512K (Unified)
CPU /sys/devices/system/cpu/cpu18 Level 3 Cache: 5118K (Unified)
CPU /sys/devices/system/cpu/cpu19 Level 1 Cache: 64K (Data)
CPU /sys/devices/system/cpu/cpu19 Level 1 Cache: 64K (Instruction)
CPU /sys/devices/system/cpu/cpu19 Level 2 Cache: 512K (Unified)
CPU /sys/devices/system/cpu/cpu19 Level 3 Cache: 5118K (Unified)
CPU /sys/devices/system/cpu/cpu2 Level 1 Cache: 64K (Data)
CPU /sys/devices/system/cpu/cpu2 Level 1 Cache: 64K (Instruction)
CPU /sys/devices/system/cpu/cpu2 Level 2 Cache: 512K (Unified)
CPU /sys/devices/system/cpu/cpu2 Level 3 Cache: 5118K (Unified)
CPU /sys/devices/system/cpu/cpu20 Level 1 Cache: 64K (Data)
CPU /sys/devices/system/cpu/cpu20 Level 1 Cache: 64K (Instruction)
CPU /sys/devices/system/cpu/cpu20 Level 2 Cache: 512K (Unified)
CPU /sys/devices/system/cpu/cpu20 Level 3 Cache: 5118K (Unified)
CPU /sys/devices/system/cpu/cpu21 Level 1 Cache: 64K (Data)
CPU /sys/devices/system/cpu/cpu21 Level 1 Cache: 64K (Instruction)
CPU /sys/devices/system/cpu/cpu21 Level 2 Cache: 512K (Unified)
CPU /sys/devices/system/cpu/cpu21 Level 3 Cache: 5118K (Unified)
CPU /sys/devices/system/cpu/cpu22 Level 1 Cache: 64K (Data)
CPU /sys/devices/system/cpu/cpu22 Level 1 Cache: 64K (Instruction)
CPU /sys/devices/system/cpu/cpu22 Level 2 Cache: 512K (Unified)
CPU /sys/devices/system/cpu/cpu22 Level 3 Cache: 5118K (Unified)
CPU /sys/devices/system/cpu/cpu23 Level 1 Cache: 64K (Data)
CPU /sys/devices/system/cpu/cpu23 Level 1 Cache: 64K (Instruction)
CPU /sys/devices/system/cpu/cpu23 Level 2 Cache: 512K (Unified)
CPU /sys/devices/system/cpu/cpu23 Level 3 Cache: 5118K (Unified)
CPU /sys/devices/system/cpu/cpu24 Level 1 Cache: 64K (Data)
CPU /sys/devices/system/cpu/cpu24 Level 1 Cache: 64K (Instruction)
CPU /sys/devices/system/cpu/cpu24 Level 2 Cache: 512K (Unified)
CPU /sys/devices/system/cpu/cpu24 Level 3 Cache: 5118K (Unified)
CPU /sys/devices/system/cpu/cpu25 Level 1 Cache: 64K (Data)
CPU /sys/devices/system/cpu/cpu25 Level 1 Cache: 64K (Instruction)
CPU /sys/devices/system/cpu/cpu25 Level 2 Cache: 512K (Unified)
CPU /sys/devices/system/cpu/cpu25 Level 3 Cache: 5118K (Unified)
CPU /sys/devices/system/cpu/cpu26 Level 1 Cache: 64K (Data)
CPU /sys/devices/system/cpu/cpu26 Level 1 Cache: 64K (Instruction)
CPU /sys/devices/system/cpu/cpu26 Level 2 Cache: 512K (Unified)
CPU /sys/devices/system/cpu/cpu26 Level 3 Cache: 5118K (Unified)
CPU /sys/devices/system/cpu/cpu27 Level 1 Cache: 64K (Data)
CPU /sys/devices/system/cpu/cpu27 Level 1 Cache: 64K (Instruction)
CPU /sys/devices/system/cpu/cpu27 Level 2 Cache: 512K (Unified)
CPU /sys/devices/system/cpu/cpu27 Level 3 Cache: 5118K (Unified)
CPU /sys/devices/system/cpu/cpu28 Level 1 Cache: 64K (Data)
CPU /sys/devices/system/cpu/cpu28 Level 1 Cache: 64K (Instruction)
CPU /sys/devices/system/cpu/cpu28 Level 2 Cache: 512K (Unified)
CPU /sys/devices/system/cpu/cpu28 Level 3 Cache: 5118K (Unified)
CPU /sys/devices/system/cpu/cpu29 Level 1 Cache: 64K (Data)
CPU /sys/devices/system/cpu/cpu29 Level 1 Cache: 64K (Instruction)
CPU /sys/devices/system/cpu/cpu29 Level 2 Cache: 512K (Unified)
CPU /sys/devices/system/cpu/cpu29 Level 3 Cache: 5118K (Unified)
CPU /sys/devices/system/cpu/cpu3 Level 1 Cache: 64K (Data)
CPU /sys/devices/system/cpu/cpu3 Level 1 Cache: 64K (Instruction)
CPU /sys/devices/system/cpu/cpu3 Level 2 Cache: 512K (Unified)
CPU /sys/devices/system/cpu/cpu3 Level 3 Cache: 5118K (Unified)
CPU /sys/devices/system/cpu/cpu30 Level 1 Cache: 64K (Data)
CPU /sys/devices/system/cpu/cpu30 Level 1 Cache: 64K (Instruction)
CPU /sys/devices/system/cpu/cpu30 Level 2 Cache: 512K (Unified)
CPU /sys/devices/system/cpu/cpu30 Level 3 Cache: 5118K (Unified)
CPU /sys/devices/system/cpu/cpu31 Level 1 Cache: 64K (Data)
CPU /sys/devices/system/cpu/cpu31 Level 1 Cache: 64K (Instruction)
CPU /sys/devices/system/cpu/cpu31 Level 2 Cache: 512K (Unified)
CPU /sys/devices/system/cpu/cpu31 Level 3 Cache: 5118K (Unified)
CPU /sys/devices/system/cpu/cpu32 Level 1 Cache: 64K (Data)
CPU /sys/devices/system/cpu/cpu32 Level 1 Cache: 64K (Instruction)
CPU /sys/devices/system/cpu/cpu32 Level 2 Cache: 512K (Unified)
CPU /sys/devices/system/cpu/cpu32 Level 3 Cache: 5118K (Unified)
CPU /sys/devices/system/cpu/cpu33 Level 1 Cache: 64K (Data)
CPU /sys/devices/system/cpu/cpu33 Level 1 Cache: 64K (Instruction)
CPU /sys/devices/system/cpu/cpu33 Level 2 Cache: 512K (Unified)
CPU /sys/devices/system/cpu/cpu33 Level 3 Cache: 5118K (Unified)
CPU /sys/devices/system/cpu/cpu34 Level 1 Cache: 64K (Data)
CPU /sys/devices/system/cpu/cpu34 Level 1 Cache: 64K (Instruction)
CPU /sys/devices/system/cpu/cpu34 Level 2 Cache: 512K (Unified)
CPU /sys/devices/system/cpu/cpu34 Level 3 Cache: 5118K (Unified)
CPU /sys/devices/system/cpu/cpu35 Level 1 Cache: 64K (Data)
CPU /sys/devices/system/cpu/cpu35 Level 1 Cache: 64K (Instruction)
CPU /sys/devices/system/cpu/cpu35 Level 2 Cache: 512K (Unified)
CPU /sys/devices/system/cpu/cpu35 Level 3 Cache: 5118K (Unified)
CPU /sys/devices/system/cpu/cpu36 Level 1 Cache: 64K (Data)
CPU /sys/devices/system/cpu/cpu36 Level 1 Cache: 64K (Instruction)
CPU /sys/devices/system/cpu/cpu36 Level 2 Cache: 512K (Unified)
CPU /sys/devices/system/cpu/cpu36 Level 3 Cache: 5118K (Unified)
CPU /sys/devices/system/cpu/cpu37 Level 1 Cache: 64K (Data)
CPU /sys/devices/system/cpu/cpu37 Level 1 Cache: 64K (Instruction)
CPU /sys/devices/system/cpu/cpu37 Level 2 Cache: 512K (Unified)
CPU /sys/devices/system/cpu/cpu37 Level 3 Cache: 5118K (Unified)
CPU /sys/devices/system/cpu/cpu38 Level 1 Cache: 64K (Data)
CPU /sys/devices/system/cpu/cpu38 Level 1 Cache: 64K (Instruction)
CPU /sys/devices/system/cpu/cpu38 Level 2 Cache: 512K (Unified)
CPU /sys/devices/system/cpu/cpu38 Level 3 Cache: 5118K (Unified)
CPU /sys/devices/system/cpu/cpu39 Level 1 Cache: 64K (Data)
CPU /sys/devices/system/cpu/cpu39 Level 1 Cache: 64K (Instruction)
CPU /sys/devices/system/cpu/cpu39 Level 2 Cache: 512K (Unified)
CPU /sys/devices/system/cpu/cpu39 Level 3 Cache: 5118K (Unified)
CPU /sys/devices/system/cpu/cpu4 Level 1 Cache: 64K (Data)
CPU /sys/devices/system/cpu/cpu4 Level 1 Cache: 64K (Instruction)
CPU /sys/devices/system/cpu/cpu4 Level 2 Cache: 512K (Unified)
CPU /sys/devices/system/cpu/cpu4 Level 3 Cache: 5118K (Unified)
CPU /sys/devices/system/cpu/cpu40 Level 1 Cache: 64K (Data)
CPU /sys/devices/system/cpu/cpu40 Level 1 Cache: 64K (Instruction)
CPU /sys/devices/system/cpu/cpu40 Level 2 Cache: 512K (Unified)
CPU /sys/devices/system/cpu/cpu40 Level 3 Cache: 5118K (Unified)
CPU /sys/devices/system/cpu/cpu41 Level 1 Cache: 64K (Data)
CPU /sys/devices/system/cpu/cpu41 Level 1 Cache: 64K (Instruction)
CPU /sys/devices/system/cpu/cpu41 Level 2 Cache: 512K (Unified)
CPU /sys/devices/system/cpu/cpu41 Level 3 Cache: 5118K (Unified)
CPU /sys/devices/system/cpu/cpu42 Level 1 Cache: 64K (Data)
CPU /sys/devices/system/cpu/cpu42 Level 1 Cache: 64K (Instruction)
CPU /sys/devices/system/cpu/cpu42 Level 2 Cache: 512K (Unified)
CPU /sys/devices/system/cpu/cpu42 Level 3 Cache: 5118K (Unified)
CPU /sys/devices/system/cpu/cpu43 Level 1 Cache: 64K (Data)
CPU /sys/devices/system/cpu/cpu43 Level 1 Cache: 64K (Instruction)
CPU /sys/devices/system/cpu/cpu43 Level 2 Cache: 512K (Unified)
CPU /sys/devices/system/cpu/cpu43 Level 3 Cache: 5118K (Unified)
CPU /sys/devices/system/cpu/cpu44 Level 1 Cache: 64K (Data)
CPU /sys/devices/system/cpu/cpu44 Level 1 Cache: 64K (Instruction)
CPU /sys/devices/system/cpu/cpu44 Level 2 Cache: 512K (Unified)
CPU /sys/devices/system/cpu/cpu44 Level 3 Cache: 5118K (Unified)
CPU /sys/devices/system/cpu/cpu45 Level 1 Cache: 64K (Data)
CPU /sys/devices/system/cpu/cpu45 Level 1 Cache: 64K (Instruction)
CPU /sys/devices/system/cpu/cpu45 Level 2 Cache: 512K (Unified)
CPU /sys/devices/system/cpu/cpu45 Level 3 Cache: 5118K (Unified)
CPU /sys/devices/system/cpu/cpu46 Level 1 Cache: 64K (Data)
CPU /sys/devices/system/cpu/cpu46 Level 1 Cache: 64K (Instruction)
CPU /sys/devices/system/cpu/cpu46 Level 2 Cache: 512K (Unified)
CPU /sys/devices/system/cpu/cpu46 Level 3 Cache: 5118K (Unified)
CPU /sys/devices/system/cpu/cpu47 Level 1 Cache: 64K (Data)
CPU /sys/devices/system/cpu/cpu47 Level 1 Cache: 64K (Instruction)
CPU /sys/devices/system/cpu/cpu47 Level 2 Cache: 512K (Unified)
CPU /sys/devices/system/cpu/cpu47 Level 3 Cache: 5118K (Unified)
CPU /sys/devices/system/cpu/cpu5 Level 1 Cache: 64K (Data)
CPU /sys/devices/system/cpu/cpu5 Level 1 Cache: 64K (Instruction)
CPU /sys/devices/system/cpu/cpu5 Level 2 Cache: 512K (Unified)
CPU /sys/devices/system/cpu/cpu5 Level 3 Cache: 5118K (Unified)
CPU /sys/devices/system/cpu/cpu6 Level 1 Cache: 64K (Data)
CPU /sys/devices/system/cpu/cpu6 Level 1 Cache: 64K (Instruction)
CPU /sys/devices/system/cpu/cpu6 Level 2 Cache: 512K (Unified)
CPU /sys/devices/system/cpu/cpu6 Level 3 Cache: 5118K (Unified)
CPU /sys/devices/system/cpu/cpu7 Level 1 Cache: 64K (Data)
CPU /sys/devices/system/cpu/cpu7 Level 1 Cache: 64K (Instruction)
CPU /sys/devices/system/cpu/cpu7 Level 2 Cache: 512K (Unified)
CPU /sys/devices/system/cpu/cpu7 Level 3 Cache: 5118K (Unified)
CPU /sys/devices/system/cpu/cpu8 Level 1 Cache: 64K (Data)
CPU /sys/devices/system/cpu/cpu8 Level 1 Cache: 64K (Instruction)
CPU /sys/devices/system/cpu/cpu8 Level 2 Cache: 512K (Unified)
CPU /sys/devices/system/cpu/cpu8 Level 3 Cache: 5118K (Unified)
CPU /sys/devices/system/cpu/cpu9 Level 1 Cache: 64K (Data)
CPU /sys/devices/system/cpu/cpu9 Level 1 Cache: 64K (Instruction)
CPU /sys/devices/system/cpu/cpu9 Level 2 Cache: 512K (Unified)
CPU /sys/devices/system/cpu/cpu9 Level 3 Cache: 5118K (Unified)
Total CPU system cache: 279871488 bytes
Computed minimum array elements needed: 127214312
Minimum array elements used: 127214312

=== Check and build stream ===

=== Testing up to 48 cores ===

-------------------------------------------------------------
STREAM version $Revision: 5.9 $
-------------------------------------------------------------
This system uses 8 bytes per DOUBLE PRECISION word.
-------------------------------------------------------------
Array size = 127214312, Offset = 0
Total memory required = 2911.7 MB.
Each test is run 10 times, but only
the *best* time for each is used.
-------------------------------------------------------------
Number of Threads requested = 1
-------------------------------------------------------------
Printing one line per active thread....
-------------------------------------------------------------
Your clock granularity/precision appears to be 1 microseconds.
Each test below will take on the order of 254582 microseconds.
    (= 254582 clock ticks)
Increase the size of the arrays if this shows that
you are not getting at least 20 clock ticks per test.
-------------------------------------------------------------
WARNING -- The above is only a rough guideline.
For best results, please be sure you know the
precision of your system timer.
-------------------------------------------------------------
Function      Rate (MB/s)   Avg time     Min time     Max time
Copy:        6057.6445       0.3418       0.3360       0.3750
Scale:       6028.3481       0.3442       0.3376       0.3786
Add:         6304.5394       0.4900       0.4843       0.5142
Triad:       6236.0693       0.4968       0.4896       0.5219
-------------------------------------------------------------
Solution Validates
-------------------------------------------------------------

Number of Threads requested = 2
Function      Rate (MB/s)   Avg time     Min time     Max time
Triad:      12471.5252       0.2448       0.2448       0.2449

Number of Threads requested = 3
Function      Rate (MB/s)   Avg time     Min time     Max time
Triad:      15952.3092       0.1914       0.1914       0.1915

Number of Threads requested = 4
Function      Rate (MB/s)   Avg time     Min time     Max time
Triad:      24935.8135       0.1225       0.1224       0.1225

Number of Threads requested = 5
Function      Rate (MB/s)   Avg time     Min time     Max time
Triad:      26223.8995       0.1165       0.1164       0.1166

Number of Threads requested = 6
Function      Rate (MB/s)   Avg time     Min time     Max time
Triad:      36886.6048       0.0828       0.0828       0.0828

Number of Threads requested = 7
Function      Rate (MB/s)   Avg time     Min time     Max time
Triad:      36930.7515       0.0827       0.0827       0.0828

Number of Threads requested = 8
Function      Rate (MB/s)   Avg time     Min time     Max time
Triad:      38068.1227       0.0826       0.0802       0.0833

Number of Threads requested = 9
Function      Rate (MB/s)   Avg time     Min time     Max time
Triad:      21442.7286       0.1506       0.1424       0.1639

Number of Threads requested = 10
Function      Rate (MB/s)   Avg time     Min time     Max time
Triad:      22577.0833       0.1356       0.1352       0.1359

Number of Threads requested = 11
Function      Rate (MB/s)   Avg time     Min time     Max time
Triad:      23312.3289       0.1311       0.1310       0.1311

Number of Threads requested = 12
Function      Rate (MB/s)   Avg time     Min time     Max time
Triad:      40323.1058       0.0760       0.0757       0.0763

Number of Threads requested = 13
Function      Rate (MB/s)   Avg time     Min time     Max time
Triad:      47004.6724       0.0652       0.0650       0.0654

Number of Threads requested = 14
Function      Rate (MB/s)   Avg time     Min time     Max time
Triad:      44424.2111       0.0687       0.0687       0.0688

Number of Threads requested = 15
Function      Rate (MB/s)   Avg time     Min time     Max time
Triad:      52259.2348       0.0585       0.0584       0.0587

Number of Threads requested = 16
Function      Rate (MB/s)   Avg time     Min time     Max time
Triad:      64229.4556       0.0476       0.0475       0.0477

Number of Threads requested = 17
Function      Rate (MB/s)   Avg time     Min time     Max time
Triad:      34654.6042       0.0969       0.0881       0.0989

Number of Threads requested = 18
Function      Rate (MB/s)   Avg time     Min time     Max time
Triad:      43236.4397       0.0846       0.0706       0.0985

Number of Threads requested = 19
Function      Rate (MB/s)   Avg time     Min time     Max time
Triad:      40173.4578       0.0783       0.0760       0.0799

Number of Threads requested = 20
Function      Rate (MB/s)   Avg time     Min time     Max time
Triad:      52418.1724       0.0585       0.0582       0.0587

Number of Threads requested = 21
Function      Rate (MB/s)   Avg time     Min time     Max time
Triad:      59309.7805       0.0517       0.0515       0.0518

Number of Threads requested = 22
Function      Rate (MB/s)   Avg time     Min time     Max time
Triad:      55953.3174       0.0547       0.0546       0.0548

Number of Threads requested = 23
Function      Rate (MB/s)   Avg time     Min time     Max time
Triad:      69792.5266       0.0439       0.0437       0.0443

Number of Threads requested = 24
Function      Rate (MB/s)   Avg time     Min time     Max time
Triad:      78001.9366       0.0393       0.0391       0.0393

Number of Threads requested = 25
Function      Rate (MB/s)   Avg time     Min time     Max time
Triad:      56661.3804       0.0670       0.0539       0.0740

Number of Threads requested = 26
Function      Rate (MB/s)   Avg time     Min time     Max time
Triad:      51899.3931       0.0624       0.0588       0.0674

Number of Threads requested = 27
Function      Rate (MB/s)   Avg time     Min time     Max time
Triad:      48560.3902       0.0681       0.0629       0.0704

Number of Threads requested = 28
Function      Rate (MB/s)   Avg time     Min time     Max time
Triad:      63773.3287       0.0485       0.0479       0.0498

Number of Threads requested = 29
Function      Rate (MB/s)   Avg time     Min time     Max time
Triad:      67561.1570       0.0456       0.0452       0.0457

Number of Threads requested = 30
Function      Rate (MB/s)   Avg time     Min time     Max time
Triad:      59568.8426       0.0514       0.0513       0.0515

Number of Threads requested = 31
Function      Rate (MB/s)   Avg time     Min time     Max time
Triad:      54612.0337       0.0565       0.0559       0.0567

Number of Threads requested = 32
Function      Rate (MB/s)   Avg time     Min time     Max time
Triad:      81013.5506       0.0378       0.0377       0.0379

Number of Threads requested = 33
Function      Rate (MB/s)   Avg time     Min time     Max time
Triad:      58938.5382       0.0570       0.0518       0.0594

Number of Threads requested = 34
Function      Rate (MB/s)   Avg time     Min time     Max time
Triad:      58142.9574       0.0555       0.0525       0.0591

Number of Threads requested = 35
Function      Rate (MB/s)   Avg time     Min time     Max time
Triad:      52356.8789       0.0590       0.0583       0.0594

Number of Threads requested = 36
Function      Rate (MB/s)   Avg time     Min time     Max time
Triad:      64303.6362       0.0481       0.0475       0.0485

Number of Threads requested = 37
Function      Rate (MB/s)   Avg time     Min time     Max time
Triad:      63251.3840       0.0483       0.0483       0.0484

Number of Threads requested = 38
Function      Rate (MB/s)   Avg time     Min time     Max time
Triad:      74401.3522       0.0411       0.0410       0.0412

Number of Threads requested = 39
Function      Rate (MB/s)   Avg time     Min time     Max time
Triad:      77623.2130       0.0394       0.0393       0.0394

Number of Threads requested = 40
Function      Rate (MB/s)   Avg time     Min time     Max time
Triad:      80152.0442       0.0383       0.0381       0.0384

Number of Threads requested = 41
Function      Rate (MB/s)   Avg time     Min time     Max time
Triad:      68952.6217       0.0443       0.0443       0.0443

Number of Threads requested = 42
Function      Rate (MB/s)   Avg time     Min time     Max time
Triad:      69971.7614       0.0437       0.0436       0.0437

Number of Threads requested = 43
Function      Rate (MB/s)   Avg time     Min time     Max time
Triad:      71488.5304       0.0428       0.0427       0.0430

Number of Threads requested = 44
Function      Rate (MB/s)   Avg time     Min time     Max time
Triad:      72992.9602       0.0419       0.0418       0.0420

Number of Threads requested = 45
Function      Rate (MB/s)   Avg time     Min time     Max time
Triad:      75000.9485       0.0408       0.0407       0.0409

Number of Threads requested = 46
Function      Rate (MB/s)   Avg time     Min time     Max time
Triad:      76208.7407       0.0401       0.0401       0.0402

Number of Threads requested = 47
Function      Rate (MB/s)   Avg time     Min time     Max time
Triad:      77969.6418       0.0392       0.0392       0.0393

Number of Threads requested = 48
Function      Rate (MB/s)   Avg time     Min time     Max time
Triad:      79731.3522       0.0384       0.0383       0.0385