data on devel code perf dip

I have an example of runs that illustrate a performance 
problem that occurred between installing the 7/18 and 8/1 
development release codes.  I'm running on a PPC64 8-way system, 
with 16GB of memory (on a 15GB virtual machine), with a DBT2 
workload configured as a 16 warehouse DBT2 with 16 db connections, 
no think time.  Using Linux 2.6.11/gentoo.


Overview:

Before picture: 
run 42 code updated from cvs July 18 12:18pm pacific
metric: 8374.71 NOTPM

After picture (same config params, db size, etc):
run 59 code downloaded from cvs (all of it) Aug1 9:58AM pacific
metric: 4787.61 NOTPM

Oprofile data in 59 (top part copied at the end of this message) but
not in 42 (although the binary is still around and I could in theory
rerun and get it).  

I'd like to first get back to the throughput of run 42, and then
remove what's keeping me from getting higher than 45% user with 
the original July 18 drop.

************************************************************
Here are more details:

(RUN 42) the previous performance 8374.71 new-order transactions per
minute (NOTPM) and %user around 45% on 8 processors.  Even this level of
user utilization is not good, as there is still some bandwidth left in
the log device (not much, but some).

http://developer.osdl.org/maryedie/DBT2_PGSQL/42/


This development code we got on July 18 12:18pm pacific. Tuned it and got
to run 42 on July 26. 

(RUN 59)  This is an example of a run with code downloaded (whole) on
Aug1 9:58am Pacific.  4787.61 new-order transactions per minute (NOTPM)
and ~22% user over 8 cpus.  This has the same config parameters as 42,
with autovacuum off.  However, I turned on oprofile for this run. (runs
without oprofile are equivalent).

http://developer.osdl.org/maryedie/DBT2_PGSQL/59/


Top things for postgreSQL in the oprofile are copied below.

CPU: ppc64 POWER5, speed 1656.38 MHz (estimated)
Counted CYCLES events (Processor cycles) with a unit mask of 0x00 (No unit mask) count 100000
samples  %        app name                 symbol name
164623113 70.5372  kernel-2.6.11.3          .shared_idle
6641569   2.8458  libc-2.3.4.so            (no symbols)
4011874   1.7190  postgres                 .AllocSetAlloc
2798355   1.1990  postgres                 .CreateLWLocks
1731687   0.7420  postgres                 .SearchCatCache
1701936   0.7292  postgres                 .MemoryContextAllocZeroAligned
1509903   0.6470  postgres                 .FunctionCall2
1320467   0.5658  postgres                 .hash_search
1275278   0.5464  oprofiled                (no symbols)
1271477   0.5448  postgres                 .yyparse
1099396   0.4711  postgres                 .OpernameGetCandidates
794802    0.3406  postgres                 .expression_tree_walker
731969    0.3136  postgres                 .MemoryContextAlloc
730266    0.3129  postgres                 ._bt_compare
709793    0.3041  postgres                 .DirectFunctionCall1
689405    0.2954  postgres                 .base_yylex
683776    0.2930  postgres                 .hash_any
635680    0.2724  postgres                 .DLMoveToFront
632098    0.2708  kernel-2.6.11.3          .___dst_free
562111    0.2409  kernel-2.6.11.3          .sys_setrlimit
508608    0.2179  postgres                 .AtEOXact_CatCache
499318    0.2139  postgres                 .heap_release_fetch
498136    0.2134  kernel-2.6.11.3          .tty_write
494020    0.2117  postgres                 .XLogInsert


-- 
Mary Edie Meredith
Initiative Manager
Open Source Development Labs
maryedie@hotmail.com
503-906-1942