Hi Mark,
Batching statements in JDBC will probably get you up to 1000 operations/sec (reading from disk, slow database) or 10000 operations/sec (reading from memory and/or fast database). I have not yet used the JDBC COPY patch, but it might be what you want, if you need higher speeds. I had always associated the COPY command with properly formatted input files, but I have read all the patch needs is an InputStream, so it might be exactly what you need.
I am planning to package pgBee into a library, so that it may be used from other java programs or become part of a workflow, but it's always going to be file based. Then again, when you say 'immense amount of data' and 'periodically' you make me worry about JVM coping with your memory requirements.
cheers,
Michael
On Thu, Oct 16, 2008 at 12:14 AM,
<mark_addleman@bigfoot.com> wrote:
Hi, Michael -
Thanks for the batch pointer. We are currently using batches with
varying sizes but still aren't getting the performance we'd like.
As for including the COPY facility in the JDBC driver versus doing
bulk loads from a command line utility: Our online application
receives an immense amount of data periodically and must store that
data into Postgres before the next bulk arrives. It makes sense to
process and store this data entirely within the confines of a JVM and
within a transaction thus COPY support in the JDBC driver is very
helpful for us.