Re: connection lost with concurrent transactions

On Mon, May 16, 2011 at 2:39 PM, Otto Vazquez <otto.vazquez@gmail.com> wrote:
> I just installed RabbitMQ in my local machine (ubuntu 10.10, postgres 8.4.8
> and rabbitmq 1.8.0, other stuff is the same version) and everything worked
> fine.
> So I tried to do the same with the dev environment (centos 5.0 final,
> rabbitmq 1.7.2 from epel repo). We have 6 machines: 2 cds (where tasks are
> executed), 2 cms and 2 db (master/slave, so only master accessible).
> I have tried different RabbitMQ configuration: just in db master host, in
> both cds hosts, in all hosts... no way. Always getting same error:
> [2011-05-16 14:20:12,084: WARNING/PoolWorker-1]
> /usr/lib/python2.6/site-packages/celery-2.2.4-py2.6.egg/celery/worker/job.py:114:
> UserWarning: Exception outside body: <class 'psycopg2.InterfaceError'>:
> connection already closed

This is different from the first error you reported, isn't it? You
were initially reporting a "no connection to server".

The last error seems more manageable: it seems connection.close() is
invoked twice. Can you trace who calls the close method before django?
You may use a connection subclass to discover this:

    class TraceConn(psycopg2.extensions.connection):
        def close(self):
            # replace with the logging strategy you need
            import traceback
            traceback.print_stack()
            psycopg2.extensions.connection.close(self)

and create your connection with

    conn = psycopg2.connect(DSN, connection_factory=TraceConn)

I don't know if django supports any way to pass the connection_factory
method. If it doesn't you may inject it via monkeypatching by running
very early (e.g. in the settings.py):

    connect_orig = psycopg2.connect

    def my_connect(dsn, **kwargs):
        kwargs['connection_factory'] = TraceConn
        return connect_orig(dsn, **kwargs)

    psycopg2.connect = my_connect


> So now, I'm not sure if this is a matter of architecture, version bug or the
> connector is not working properly.

I don't know what celery does.


> BTW, we are not using any db middleware (pgpool or pgbouncer)

Good to know.


> Any hint before moving to MySQL?

Can you check if MySQLdb's connections support close() twice? Maybe
the difference is there (different implementations of the DBAPI
requirement "The connection will be unusable from [close()] forward".


-- Daniele