Re: 60 core performance with 9.3

On 11/07/14 20:22, Andres Freund wrote:
> On 2014-07-11 12:40:15 +1200, Mark Kirkwood wrote:
>> Full report http://paste.ubuntu.com/7777886/
>
>> #
>>       8.82%        postgres  [kernel.kallsyms]        [k]
>> _raw_spin_lock_irqsave
>>                    |
>>                    --- _raw_spin_lock_irqsave
>>                       |
>>                       |--75.69%-- pagevec_lru_move_fn
>>                       |          __lru_cache_add
>>                       |          lru_cache_add
>>                       |          putback_lru_page
>>                       |          migrate_pages
>>                       |          migrate_misplaced_page
>>                       |          do_numa_page
>>                       |          handle_mm_fault
>>                       |          __do_page_fault
>>                       |          do_page_fault
>>                       |          page_fault
>
> So, the majority of the time is spent in numa page migration. Can you
> disable numa_balancing? I'm not sure if your kernel version does that at
> runtime or whether you need to reboot.
> The kernel.numa_balancing sysctl might work. Otherwise you probably need
> to boot with numa_balancing=0.
>
> It'd also be worthwhile to test this with numactl --interleave.
>

Trying out with numa_balancing=0 seemed to get essentially the same
performance. Similarly wrapping postgres startup with --interleave.

All this made me want to try with numa *really* disabled. So rebooted
the box with "numa=off" appended to the kernel cmdline. Somewhat
surprisingly (to me anyway), the numbers were essentially identical. The
profile, however is quite different:

Full report at http://paste.ubuntu.com/7806285/


      4.56%         postgres  [kernel.kallsyms]         [k]
_raw_spin_lock_irqsave


                    |
                    --- _raw_spin_lock_irqsave
                       |
                       |--41.89%-- try_to_wake_up
                       |          |
                       |          |--96.12%-- default_wake_function
                       |          |          |
                       |          |          |--99.96%-- pollwake
                       |          |          |          __wake_up_common
                       |          |          |          __wake_up_sync_key
                       |          |          |          sock_def_readable
                       |          |          |          |
                       |          |          |          |--99.94%--
unix_stream_sendmsg
                       |          |          |          |
sock_sendmsg
                       |          |          |          |
SYSC_sendto
                       |          |          |          |
sys_sendto
                       |          |          |          |          tracesys
                       |          |          |          |
__libc_send
                       |          |          |          |          pq_flush
                       |          |          |          |
ReadyForQuery
                       |          |          |          |
PostgresMain
                       |          |          |          |
ServerLoop
                       |          |          |          |
PostmasterMain
                       |          |          |          |          main
                       |          |          |          |
__libc_start_main
                       |          |          |           --0.06%-- [...]
                       |          |           --0.04%-- [...]
                       |          |
                       |          |--2.87%-- wake_up_process
                       |          |          |
                       |          |          |--95.71%--
wake_up_sem_queue_do
                       |          |          |          SYSC_semtimedop
                       |          |          |          sys_semop
                       |          |          |          tracesys
                       |          |          |          __GI___semop
                       |          |          |          |
                       |          |          |          |--99.75%--
LWLockRelease
                       |          |          |          |          |

                       |          |          |          |
|--25.09%-- RecordTransactionCommit
                       |          |          |          |          |
       CommitTransaction
                       |          |          |          |          |
       CommitTransactionCommand
                       |          |          |          |          |
       finish_xact_command.part.4
                       |          |          |          |          |
       PostgresMain
                       |          |          |          |          |
       ServerLoop
                       |          |          |          |          |
       PostmasterMain
                       |          |          |          |          |
       main
                       |          |          |          |          |
       __libc_start_main



regards

Mark