Re: [HACKERS] Active zombies at AIX

Last update on the problem.
Using kdb tool (thank's to Tony Reix for advice and help) we get the following trace of Poastgres backend in existing stack trace:

pvthread+073000 STACK:
[005E1958]slock+000578 (00000000005E1958, 8000000000001032 [??])
[00009558].simple_lock+000058 ()
[00651DBC]vm_relalias+00019C (??, ??, ??, ??, ??)
[006544AC]vm_map_entry_delete+00074C (??, ??, ??)
[00659C30]vm_map_delete+000150 (??, ??, ??, ??)
[00659D88]vm_map_deallocate+000048 (??, ??)
[0011C588]kexitx+001408 (??)
[000BB08C]kexit+00008C ()
___ Recovery (FFFFFFFFFFF9290) ___
WARNING: Eyecatcher/version mismatch in RWA


So there seems to be lock contention while unmapping memory segments.
My assumption was that Postgres is detaching all attached segments before exit (in shmem_exit callback or earlier).
I have added logging around proc_exit_prepare function (which is called by atexit callback) and check that it completes immediately.
So I thought that this vm_map_deallocate can be related with deallocation of normal (malloced) memory, because in Linux memory allocator may use mmap.
But in AIX it is not true.
Below is report of Bergamini Demien (once again a lot of thanks  for help with investigation the problem):

The memory allocator in AIX libc does not use mmap and vm_relalias() is only called for shared memory mappings.

I talked with the AIX VMM expert at IBM and he said that what you hit is one of the most common performance bottlenecks in AIX memory management.

He also said that SysV Shared Memory (shmget/shmat) perform better on AIX than mmap.

Some improvements have been made in AIX 6.1 (see “perf suffers when procs sharing the same segs all exit at once”: http://www-01.ibm.com/support/docview.wss?uid=isg1IZ83819) but it does not help in your case.

In src/backend/port/sysv_shmem.c, it says that PostgreSQL 9.3 switched from using SysV Shared Memory to using mmap.

Maybe you could try to switch back to using SysV Shared Memory on AIX to see if it helps performance-wise.

Also, the good news is that there are some restricted tunables in AIX that can be tweaked to help different workloads which may have different demands.

One of them is relalias_percentage which works with force_relalias_lite:

# vmo -h relalias_percentage

Help for tunable relalias_percentage:

Purpose:

If force_relalias_lite is set to 0, then this specifies the factor used in the heuristic to decide whether to avoid locking the source mmapped segment or not.

Values:

        Default: 0
        Range: 0 - 32767
        Type: Dynamic
        Unit:

Tuning:

This is used when tearing down an mmapped region and is a scalability statement, where avoiding the lock may help system throughput, but, in some cases, at the cost of more compute time used. If the number of pages being unmapped is less than this value divided by 100 and multiplied by the total number of pages in memory in the source mmapped segment, then the source lock will be avoided. A value of 0 for relalias_percentage, with force_relalias_lite also set to 0, will cause the source segment lock to always be taken. Effective values for relalias_percentage will vary by workload, however, a suggested value is: 200.

 
You may also try to play with the munmap_npages vmo tunable.
Your vmo settings for lgpg_size, lgpg_regions and v_pinshm already seem correct.

Hi hackers,

Yet another story about AIX. For some reasons AIX very slowly cleaning zombie processes.
If we launch pgbench with -C parameter then very soon limit for maximal number of connections is exhausted.
If maximal number of connection is set to 1000, then after ten seconds of pgbench activity we get about 900 zombie processes and it takes about 100 seconds (!)
before all of them are terminated.

proctree shows a lot of defunt processes:

[14:44:41]root@postgres:~ # proctree 26084446
26084446 /opt/postgresql/xlc/9.6/bin/postgres -D /postg_fs/postgresql/xlc
4784362 <defunct>
4980786 <defunct>
11403448 <defunct>
11468930 <defunct>
11993176 <defunct>
12189710 <defunct>
12517390 <defunct>
13238374 <defunct>
13565974 <defunct>
13893826 postgres: wal writer process
14024716 <defunct>
15401000 <defunct>
...
25691556 <defunct> But ps shows that status of process is <existing>

[14:46:02]root@postgres:~ # ps -elk | grep 25691556

• A - 25691556 - - - - - <exiting>

Breakpoint set in reaper() function in postmaster shows that each invocation of this functions (called by SIGCHLD handler) proceed 5-10 PIDS per invocation.
So there are two hypothesis: either AIX is very slowly delivering SIGCHLD to parent, either exit of process takes too much time.

The fact the backends are in exiting state makes second hypothesis more reliable.
We have tried different Postgres configurations with local and TCP sockets, with different amount of shared buffers and built both with gcc and xlc.
In all cases behavior is similar: zombies do not want to die.
As far as it is not possible to attach debugger to defunct process, it is not clear how to understand what's going on.

I wonder if somebody has encountered similar problems at AIX and may be can suggest some solution to solve this problem.
Thanks in advance

-- 
Konstantin Knizhnik
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company 

-- 
Konstantin Knizhnik
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company