Further info on LWLock behavior

I added some code to the backend to count the number of LWLockAcquire
calls and the number of resultant process blocks (IpcSemaphoreLocks)
across all PG processes.  Here are the results for a pgbench run with
scale factor 500, number of clients 10, proposed LWLock patch installed
on that 4-way Linux box.

LWLock        acquires    blocks        fraction

BufMgrLock    1238892        60273        0.0486507298457008
LockMgrLock    690150        83250        0.120625950880243
OidGenLock    10004        3        0.000299880047980808
XidGenLock    60019        29        0.000483180326230027
ShmemIndexLock    180        0        0
SInvalLock    817017        1940        0.00237449159564611
FreeSpaceLock    290        0        0
WALInsertLock    80039        6139        0.0767001086970102
WALWriteLock    10194        1180        0.115754365312929
ControlFileLock    141        0        0
CLogControlLock    25366        56        0.0022076795710794
buf cxt locks    930006        172        0.000184945043365312
buf io locks    31859        142        0.00445713926990803
clog buf locks    1880        2        0.00106382978723404

Interesting data, eh?  In particular, it seems my previous opinion
that BufMgrLock was the main issue is all wet: the LockMgrLock accounts
for more blockages despite being locked fewer times.  AFAICS this must
mean that the average time of holding LockMgrLock is larger than the
average time of holding BufMgrLock, and that we ought to look at how
to reduce that.  The WAL locks also seem to have disproportionately
large blocking percentages.
        regards, tom lane