Found the culprit, the app had thrown locks on the first table in the replicate set and it was waiting for those locks to be released.
From: Plugge, Joe R.
Sent: Wednesday, June 23, 2010 12:23 PM
To: pgsql-admin@postgresql.org
Subject: Slony - I question
I am using slony1-2.0.3 and needed to make some wholesale changes, so like other times I stopped all slon processes on my master box as well as on the 2 slave machines. I then dropped the slony schema on all three, completely removing it, made my changes. I then ran my slonik script at the master, started up all of the slon processes and ran my subscribe script on both slaves. My problem, is that only one of the slaves got the copy and synched, the other one just hangs at this point:
2010-06-23 03:43:18 EDT CONFIG storeListen: li_origin=1 li_receiver=3 li_provider=1
2010-06-23 03:43:18 EDT CONFIG storeListen: li_origin=2 li_receiver=3 li_provider=1
2010-06-23 03:43:18 EDT CONFIG storeListen: li_origin=1 li_receiver=3 li_provider=1
2010-06-23 03:43:18 EDT CONFIG storeListen: li_origin=2 li_receiver=3 li_provider=1
2010-06-23 03:43:18 EDT CONFIG storeListen: li_origin=1 li_receiver=3 li_provider=1
2010-06-23 03:43:18 EDT CONFIG storeListen: li_origin=2 li_receiver=3 li_provider=1
2010-06-23 03:43:19 EDT CONFIG remoteWorkerThread_1: update provider configuration
2010-06-23 03:44:01 EDT CONFIG storeSubscribe: sub_set=1 sub_provider=1 sub_forward='f'
NOTICE: subscribe set: omit_copy=f
2010-06-23 03:44:02 EDT CONFIG storeListen: li_origin=1 li_receiver=3 li_provider=1
2010-06-23 03:44:02 EDT CONFIG storeListen: li_origin=2 li_receiver=3 li_provider=1
2010-06-23 03:44:02 EDT INFO copy_set 1 - omit=f - bool=0
2010-06-23 03:44:02 EDT INFO omit is FALSE
2010-06-23 03:44:02 EDT CONFIG version for "dbname=holly host=linux2276 port=5432 user=slony" is 80401
2010-06-23 03:44:02 EDT CONFIG remoteWorkerThread_1: connected to provider DB
2010-06-23 03:44:02 EDT CONFIG remoteWorkerThread_1: prepare to copy table "public"."hlm_owners"
I query the pg_stat_activity table and see that the table in question has been locked ....
I am using postgres 8.4.1. This worked many times before, so I am wondering if I am running into some new condition.
Just wondering what else to possible check for?
joe