Every morning for the past several days, when I come in to work, I find the postgres process running at 100% CPU and the logs filled with entries every minute with the warning "PDTWARNING: worker took too long to start; canceled"
picasso:~ postrgres$ ps -ef | grep post
642 69200 69199 0 Tue08AM ?? 0:01.61 postgres: logger process
642 69202 69199 0 Tue08AM ?? 0:07.27 postgres: writer process
642 69203 69199 0 Tue08AM ?? 0:05.17 postgres: wal writer process
642 69204 69199 0 Tue08AM ?? 0:02.48 postgres: autovacuum launcher process
642 69205 69199 0 Tue08AM ?? 0:03.71 postgres: stats collector process
642 69226 69199 0 Tue08AM ?? 0:00.01 postgres: postgres postgres [local] idle
642 69247 69199 0 9:04AM ?? 0:00.05 postgres: postgres postgres ::1(58621) idle
642 70284 69199 0 11:38AM ?? 0:00.00 (postgres)
642 70286 69199 0 11:38AM ?? 0:00.00 (postgres)
642 70290 69199 0 11:39AM ?? 0:00.00 (postgres)
642 72214 69199 0 6:13PM ?? 0:00.00 (postgres)
642 72232 69199 0 6:18PM ?? 0:00.00 (postgres)
642 72233 69199 0 6:18PM ?? 0:00.00 (postgres)
642 69199 1 0 Tue08AM ttys006 881:48.66 /usr/local/pgsql-9.1/bin/postgres -D data
I am unable to establish new connections to the database. New command line psql requests hang effectively forever, as does trying to connect with pgadmin. Existing psql sessions remain active and responsive, unless I try to connect to a different database.
The only recourse has been to kill -9 the postgres process and restart.
I am not familiar with strace (dtrace on OSX, I guess...) so I'm not sure what I'm looking for.
I did do % dtrace -P syscall -p 69199
and I notice now that postgres is no longer using 100% of the cpu, but I am still unable to establish new connections. And the warning messages are still being written to the log every minute.
The only thing that I can think of that has changed recently is that I have been experimenting with container-managed security (FORM-based authentication) using Glassfish 3.1.2.
Apologies if this was answered somewhere and I haven't found it.
Thanks,
Linus