Thread: Hyperthreading (was: Two identical systems, radically different performance)

I've confirmed that hyperthreading causes a huge drop in performance on a 2x4-core Intel Xeon E5620 2.40GHz system.  The bottom line is:

  ~3200 TPS max with hyperthreading
  ~9000 TPS max without hyprethreading

Here are my results.

"Hyprethreads" (Run1) is "out of the box" with hyperthreads enabled. Only one column of hyperthread results are shown, but prior to today's work I ran this a dozen or so times, and "Run1" is very representative of all those runs.  I also rebooted and confirmed that rebooting had no effect.

"+NoHyperthreads" (Run2-5) involved rebooting into the BIOS and disabling hyperthreads.

"+NoAutoVacuum" (Run6-8) means I turned off autovacuum in the postgresql.conf file and restarted Postgres (with hyperthreads still disabled).

          Hyperthreads     +NoHyperthreads         +NoAutoVacuum
          ------------  ----------------------   ----------------
-c  -t        Run1      Run2  Run3  Run4  Run5   Run6  Run7  Run8
5   20000     2733      2152  2352  2398  2769   2767  2777  4463
10  10000     2783      2404  3529  3365  4397   4457  4217  4172
20  5000      3241      3128  3728  5170  5253   5252  4832  8123
30  3333      2987      5699  6180  8173  8259   6435  8225  8123
40  2500      2739      7133  6507  9298  7845   9133  9298  9230
50  2000      2119      5420  5020  8411  5670   9344  7624  8304

I'll be upgrading to 8.4.14 and making more changes to postgres.conf based on feedback.  The server is available for a day or so for more tests if anyone has suggestions.

Here's how I got these results:

su postgres
unset LANG
export LD_LIBRARY_PATH=/usr/local/pgsql/lib
/usr/local/pgsql/bin/initdb --pgdata=/data/postgres/main \
  --xlogdir=/postgres_xlog/xlog --username=postgres

Edit config file:

    max_connections = 500
    shared_buffers = 1000MB
    work_mem = 128MB
    synchronous_commit = off
    full_page_writes = off
    wal_buffers = 256kB
    checkpoint_segments = 30
    effective_cache_size = 4GB
    track_activities = on
    track_counts = on
    track_functions = none
    autovacuum = on
    autovacuum_naptime = 5min
    escape_string_warning = off

createuser -U postgres test
createdb -U postgres -O test test

pgbench -i -s 100 -U test
for p in "5 20000" "10 10000" "20 5000" "30 3333" "40 2500" "50 2000" ; do
  echo
  c=`echo $p | cut -d' ' -f1`
  t=`echo $p | cut -d' ' -f2`
  cmd="pgbench -U test -c $c -t $t"
  echo "--------- $cmd ---------"
  $cmd
done

The hardware:

    CPU: 2x4-core Intex Xeon E5620 2.40 GHz

  Memory: 12 GB DDR EC

  Disks: 12x500GB disks (Western Digital 7200RPM SATA)
    XLOG: 2 disks, RAID1 ext2
    PGDATA: 8 disks, RAID10

  3WARE 9650SE-12ML with battery-backed cache.  The admin tool (tw_cli)
  indicates that the battery is charged and the cache is working on both units.

  Linux: 2.6.32-41-server #94-Ubuntu SMP (new server's disk was
  actually cloned from old server).

  Postgres: 8.4.4

Craig

Craig James <cjames@emolecules.com> writes:
> I've confirmed that hyperthreading causes a huge drop in performance on a
> 2x4-core Intel Xeon E5620 2.40GHz system.  The bottom line is:

Interesting.

> I'll be upgrading to 8.4.14 and making more changes to postgres.conf based
> on feedback.  The server is available for a day or so for more tests if
> anyone has suggestions.

It would be nice to see similar tests done with 9.2.  8.4 is kind of old
news as far as server performance is concerned.

            regards, tom lane

On 10/09/2012 03:12 PM, Craig James wrote:

>    ~3200 TPS max with hyperthreading
>    ~9000 TPS max without hyprethreading

That's really odd. We got almost the opposite effect on our X5645's.

Also, there's no way your RAID is sustaining 9000TPS. Something here
sounds fishy.

--
Shaun Thomas
OptionsHouse | 141 W. Jackson Blvd. | Suite 500 | Chicago IL, 60604
312-444-8534
sthomas@optionshouse.com

______________________________________________

See http://www.peak6.com/email_disclaimer/ for terms and conditions related to this email

On 10/9/12 1:12 PM, Craig James wrote:
> I've confirmed that hyperthreading causes a huge drop in performance on a
> 2x4-core Intel Xeon E5620 2.40GHz system.  The bottom line is:
>
>   ~3200 TPS max with hyperthreading
>   ~9000 TPS max without hyprethreading

Oh, interesting!  This is the first time I've seen results like this on
a later Linux kernel and processor.  I thought Intel had licked the
"hyperthreading penalty".

--
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com

On Tue, Oct 9, 2012 at 1:56 PM, Shaun Thomas <sthomas@optionshouse.com> wrote:
> On 10/09/2012 03:12 PM, Craig James wrote:
>
>>    ~3200 TPS max with hyperthreading
>>    ~9000 TPS max without hyprethreading
>
>
> That's really odd. We got almost the opposite effect on our X5645's.
>
> Also, there's no way your RAID is sustaining 9000TPS. Something here sounds
> fishy.

synchronous-commit=off

For your amusement ... I upgraded from 8.4.4 to 9.2.1 results. Threw away the DB completely and did a new init.  Same hardware, postgres.conf and Linux as before.

ra is "blockdev --getra" (both PGDATA and XLOG disks)
walb is postgres.conf "wal_buffers"

              ra:8192 walb:1M   ra:256 walb:1M    ra:256 walb:256kB
              ----------------  ----------------  -----------------
-c  -t        Run1  Run2  Run3  Run4  Run5  Run6  Run7  Run8  Run9
5   20000     1963  2103  2145  2292  2312  2353  2296  2175  2294
10  10000     2587  2749  2762  3252  3265  3276  3267  3228  3263
20  5000      3449  3578  3438  4910  4958  4949  4949  4927  4943
30  3333      4222  3731  3992  6929  6924  6562  6754  6995  6869
40  2500      4261  3722  4243  9286  9240  5712  9310  8530  8872
50  2000      4138  4399  3865  9213  9351  9578  8011  7651  8362

Unfortunately, distance prevents me from going to the co-location facility and trying this with hyperthreading turned back on.

Craig

On 10/09/2012 06:30 PM, Craig James wrote:

>                ra:8192 walb:1M   ra:256 walb:1M    ra:256 walb:256kB
>                ----------------  ----------------  -----------------
> -c  -t        Run1  Run2  Run3  Run4  Run5  Run6  Run7  Run8  Run9
> 40  2500      4261  3722  4243  9286  9240  5712  9310  8530  8872
> 50  2000      4138  4399  3865  9213  9351  9578  8011  7651  8362

I think I speak for more than a few people here when I say: wat.

About the only thing I can ask, is: did you make these tests fair? And
by fair, I mean:

echo 3 > /proc/sys/vm/drop_caches
pg_ctl -D /your/pg/dir restart

Between every test to make sure shared buffers and the OS inode cache
was empty before the start of each test? If you're using that bash-style
for-loop you attached earlier, probably not. Still though, I don't think
that would account for this much variance between having read-ahead at
8M as opposed to 256kb.

My head hurts.

--
Shaun Thomas
OptionsHouse | 141 W. Jackson Blvd. | Suite 500 | Chicago IL, 60604
312-444-8534
sthomas@optionshouse.com

______________________________________________

See http://www.peak6.com/email_disclaimer/ for terms and conditions related to this email

On Wed, Oct 10, 2012 at 9:52 AM, Shaun Thomas <sthomas@optionshouse.com> wrote:
> On 10/09/2012 06:30 PM, Craig James wrote:
>
>>                ra:8192 walb:1M   ra:256 walb:1M    ra:256 walb:256kB
>>                ----------------  ----------------  -----------------
>> -c  -t        Run1  Run2  Run3  Run4  Run5  Run6  Run7  Run8  Run9
>> 40  2500      4261  3722  4243  9286  9240  5712  9310  8530  8872
>> 50  2000      4138  4399  3865  9213  9351  9578  8011  7651  8362
>
>
> I think I speak for more than a few people here when I say: wat.
>
> About the only thing I can ask, is: did you make these tests fair? And by
> fair, I mean:
>
> echo 3 > /proc/sys/vm/drop_caches
> pg_ctl -D /your/pg/dir restart

Yes, I was thinking the same. Especially if you check the tendency to
perform better in higher-numbered runs. But, as you said, that doesn't
explain that jump to twice the TPS. I was thinking, and I'm not
pgbench expert, could it be that the database grows from run to run,
changing performance characteristics?

> My head hurts.

I'm just confused. No headache yet.

But really interesting numbers in any case. It these results are on
the level, then maybe the kernel's read-ahead algorithm isn't as
fool-proof as we thought? Gotta read the source. BRB

On Wed, Oct 10, 2012 at 11:44:50AM -0300, Claudio Freire wrote:
> On Wed, Oct 10, 2012 at 9:52 AM, Shaun Thomas <sthomas@optionshouse.com> wrote:
> > On 10/09/2012 06:30 PM, Craig James wrote:
> >
> >>                ra:8192 walb:1M   ra:256 walb:1M    ra:256 walb:256kB
> >>                ----------------  ----------------  -----------------
> >> -c  -t        Run1  Run2  Run3  Run4  Run5  Run6  Run7  Run8  Run9
> >> 40  2500      4261  3722  4243  9286  9240  5712  9310  8530  8872
> >> 50  2000      4138  4399  3865  9213  9351  9578  8011  7651  8362
> >
> >
> > I think I speak for more than a few people here when I say: wat.
> >
> > About the only thing I can ask, is: did you make these tests fair? And by
> > fair, I mean:
> >
> > echo 3 > /proc/sys/vm/drop_caches
> > pg_ctl -D /your/pg/dir restart
>
> Yes, I was thinking the same. Especially if you check the tendency to
> perform better in higher-numbered runs. But, as you said, that doesn't
> explain that jump to twice the TPS. I was thinking, and I'm not
> pgbench expert, could it be that the database grows from run to run,
> changing performance characteristics?
>
> > My head hurts.
>
> I'm just confused. No headache yet.
>
> But really interesting numbers in any case. It these results are on
> the level, then maybe the kernel's read-ahead algorithm isn't as
> fool-proof as we thought? Gotta read the source. BRB

Well, I have exactly the same setup here:

    new: 2x4-core Intex Xeon E5620 2.40 GHz

Let me know if you want any tests run, on SSDs or magnetic disk.  I do
have hyperthreading enabled, and Greg Smith benchmarked my server and
said it was good.

--
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

  + It's impossible for everything to be true. +

Sent this to Claudio rather than the whole list ... here it is.


Big numbers, little numbers ... I'm just reporting what pgbench tells me and how I got them.  I'm good at chemical databases, you guys are the Postgres performance experts.

On Wed, Oct 10, 2012 at 11:44 AM, Claudio Freire <klaussfreire@gmail.com> wrote:
> On Wed, Oct 10, 2012 at 9:52 AM, Shaun Thomas <sthomas@optionshouse.com> wrote:
>> On 10/09/2012 06:30 PM, Craig James wrote:
>>
>>>                ra:8192 walb:1M   ra:256 walb:1M    ra:256 walb:256kB
>>>                ----------------  ----------------  -----------------
>>> -c  -t        Run1  Run2  Run3  Run4  Run5  Run6  Run7  Run8  Run9
>>> 40  2500      4261  3722  4243  9286  9240  5712  9310  8530  8872
>>> 50  2000      4138  4399  3865  9213  9351  9578  8011  7651  8362
>>
...
> But really interesting numbers in any case. It these results are on
> the level, then maybe the kernel's read-ahead algorithm isn't as
> fool-proof as we thought? Gotta read the source. BRB

So, I've been digging.

Newer kernels (above 2.6.23) have a new read-ahead algorithm, called
the "on-demand" read ahead. Benchmarks on this new code[0] suggest
there is a penalty for random writes that wasn't there before. This
penalty is small in the benchmarks (less than 10% in all cases), but I
would imagine the issue would be amplified in the case of a kernel
with a misconfigured bg_writer (this case, IIRC).

This makes sense, in fact. Back when I tested the 8MB read-ahead on my
server, it was using 2.6.18. Now, 2.6.23 is in fact rather old, so
those benchmarks may no longer be relevant. There are tons of commits
since then[1], though only one pertaining writes from what I can tell.
However, I'm inclined to blame the bg_writer. How about tweaking the
dirty_background_ratio and dirty_ratio and re-trying?


[0] http://lwn.net/Articles/235181/
[1] https://github.com/torvalds/linux/commits/f5a246eab9a268f51ba8189ea5b098a1bfff200e/mm/readahead.c?page=1