Socket Timeouts and fatal errors (please help)

First let me divulge the pertinent info:

database server:
-----------------------------
Linux 2.6.8.1
postgres 8.1.0

webserver:
-------------------
Linux 2.6.8.1
Apache 2.0.53, mod_jk, tomcat 5.0.28
jdbc-driver 8.1-404.jdbc3
libpq.so.4.1

We have Java client objects using jdbc, some JNI C code using libpq
and some standalone c applications using libpq.
Both Java and C apps are using connection pools.  we have our max
connections set to 100 in postgres.  Our max pool size is 20 in the
java realm and 10 in the C side.

Our production server is not used heavily at the moment.  What we are
experiencing is the following:

(java side)
Sporadic (once or twice a month) Socket timeout exceptions like this:
The call in question is simply doing some data cleanup when a user
session expires.  Don't the the word 'Lock' worry you ... its all
data.  This call failed on a query.

org.postgresql.util.PSQLException: An I/O error occured while sending
to the backend.
        at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:214)
        at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:430)
        at org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:332)
        at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:324)
        at com.capecomputing.mensana.dataaccess.MensanaConnection.getConnection(MensanaConnection.java:116)
        at com.capecomputing.mensana.dataaccess.MensanaConnection.<init>(MensanaConnection.java:83)
        at
com.capecomputing.mensana.dataaccess.DatabaseUpdateAction.getEncryptedParameters(DatabaseUpdateAction.java:65)
        at
com.capecomputing.mensana.dataaccess.DatabaseUpdateAction.createEncryptedParamterList(DatabaseUpdateAction.java:39)
        at
com.capecomputing.mensana.dataaccess.CleanupLockedPrepaidClientInvoicesAction.<init>(CleanupLockedPrepaidClientInvoicesAction.java:33)
        at
com.capecomputing.mensana.dataaccess.MensanaDataAccess.cleanupLockedPrepaidInvoices(MensanaDataAccess.java:1685)
        at
com.capecomputing.mensana.webportal.application.SessionEventListener.cleanupLockedPrepaidInvoices(SessionEventListener.java:62)
        ...
Caused by: java.net.SocketException: Connection timed out
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.read(SocketInputStream.java:129)
        at java.io.BufferedInputStream.fill(BufferedInputStream.java:183)
        at java.io.BufferedInputStream.read(BufferedInputStream.java:201)
        at org.postgresql.core.PGStream.ReceiveChar(PGStream.java:254)
        at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1163)
        at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:188)
        ... 49 more


On the JNI C side the error was like this:  We retrieved a connection
from the pool.  Checked its status it was good.  Performed a query an
checked the result status and all that was returned was a fatal error.
 The JNI C side tried another connection from the pool and made
another query, this one caused a tomcat crash (the query dumped this
hs_err file)

An unexpected exception has been detected in native code outside the VM.
Unexpected Signal : 11 occurred at PC=0x400EA4D3
Function=strlen+0x33
Library=/lib/libc.so.6
...

The last message in the log was "about to call PQexec()"

Also in the standalone C application we experienced the same fatal
error message.

One of the days when we noticed the java socket timeouts we had a
power hit.  We chalked it up to that but the last one was not.  This
never happens in our non prod environments ... probably because we are
always bouncing the apps at least once a week for tweaks and tests.

Does anyone have any ideas?

I will post a sperate issue/question that may or may not be related to
the issues I stated here.  It will be titled pgadmin3 and tomcat
hangs.

thanks in advance

Doug







--
What profits a man if he gains the whole world yet loses his soul?