Re: 60 core performance with 9.3

On 2014-07-11 12:40:15 +1200, Mark Kirkwood wrote:
> On 01/07/14 22:13, Andres Freund wrote:
> >On 2014-07-01 21:48:35 +1200, Mark Kirkwood wrote:
> >>- cherry picking the last 5 commits into 9.4 branch and building a package
> >>from that and retesting:
> >>
> >>Clients | 9.4 tps 60 cores (rwlock)
> >>--------+--------------------------
> >>6       |  70189
> >>12      | 128894
> >>24      | 233542
> >>48      | 422754
> >>96      | 590796
> >>192     | 630672
> >>
> >>Wow - that is more like it! Andres that is some nice work, we definitely owe
> >>you some beers for that :-) I am aware that I need to retest with an
> >>unpatched 9.4 src - as it is not clear from this data how much is due to
> >>Andres's patches and how much to the steady stream of 9.4 development. I'll
> >>post an update on that later, but figured this was interesting enough to
> >>note for now.
> >
> >Cool. That's what I like (and expect) to see :). I don't think unpatched
> >9.4 will show significantly different results than 9.3, but it'd be good
> >to validate that. If you do so, could you post the results in the
> >-hackers thread I just CCed you on? That'll help the work to get into
> >9.5.
>
> So we seem to have nailed read only performance. Going back and revisiting
> read write performance finds:
>
> Postgres 9.4 beta
> rwlock patch
> pgbench scale = 2000
>
> max_connections = 200;
> shared_buffers = "10GB";
> maintenance_work_mem = "1GB";
> effective_io_concurrency = 10;
> wal_buffers = "32MB";
> checkpoint_segments = 192;
> checkpoint_completion_target = 0.8;
>
> clients  | tps (32 cores) | tps
> ---------+----------------+---------
> 6        |   8313         |   8175
> 12       |  11012         |  14409
> 24       |  16151         |  17191
> 48       |  21153         |  23122
> 96       |  21977         |  22308
> 192      |  22917         |  23109

On that scale - that's bigger than shared_buffers IIRC - I'd not expect
the patch to make much of a difference.

> kernel.sched_autogroup_enabled=0
> kernel.sched_migration_cost_ns=5000000
> net.core.somaxconn=1024
> /sys/kernel/mm/transparent_hugepage/enabled [never]
>
> Full report http://paste.ubuntu.com/7777886/

> #
>      8.82%        postgres  [kernel.kallsyms]        [k]
> _raw_spin_lock_irqsave
>                   |
>                   --- _raw_spin_lock_irqsave
>                      |
>                      |--75.69%-- pagevec_lru_move_fn
>                      |          __lru_cache_add
>                      |          lru_cache_add
>                      |          putback_lru_page
>                      |          migrate_pages
>                      |          migrate_misplaced_page
>                      |          do_numa_page
>                      |          handle_mm_fault
>                      |          __do_page_fault
>                      |          do_page_fault
>                      |          page_fault

So, the majority of the time is spent in numa page migration. Can you
disable numa_balancing? I'm not sure if your kernel version does that at
runtime or whether you need to reboot.
The kernel.numa_balancing sysctl might work. Otherwise you probably need
to boot with numa_balancing=0.

It'd also be worthwhile to test this with numactl --interleave.

Greetings,

Andres Freund

--
 Andres Freund                       http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services