On Sun, 24 Feb 2008 00:43:18 +0000
"Heikki Linnakangas" <heikki@enterprisedb.com> wrote:
> Incidentally, I've been working on a patch to speed up CopyReadLine.
> I was going to run some more tests first, but since we're talking
> about it, I guess I should just post the patch. I'll post to
> pgsql-patches shortly.
>
On your post to patches you mentioned only about a 5% improvement.
Don't get me wrong, 5% is 5% and I respect it greatly but as far as I
can tell we are about 300% behind the curve.
My tests were maxing out at ~22G an hour. On hardware that can do
in 360G an hour and that is assuming > 50% overhead between OS, libs,
etc... I have no choice but to conclude we have a much, much deeper and
fundamental issue going on with COPY. I am inspired by Itagaki Takahiro
and his batch update of indexes which should help greatly overall but
doesn't help my specific issue.
Forgive me for not being a C programmer and Alvaro is not online so I
would vet these questions with him first.
I know that copy is in theory a bulk loader but, when performing the
readline how many lines are we reading? Do we read up to 8192? Or do we
shove in say 8megs of data before we invoke DoCopy?
I am just curious if there is some simple low hanging fruit that is
possibly missing.
Sincerely,
Joshua D. Drake
--
The PostgreSQL Company since 1997: http://www.commandprompt.com/
PostgreSQL Community Conference: http://www.postgresqlconference.org/
Donate to the PostgreSQL Project: http://www.postgresql.org/about/donate
PostgreSQL SPI Liaison | SPI Director | PostgreSQL political pundit