Обсуждение: Possible stability issue: permanent hang on dropdb
Just had a very odd thing happen with an 8.4.2 database, and am concerned it's an undiagnosed bug: (1) created and dropped the same database 10-15 times in the course of running some tests. (2) at one point, dropdb hung. Indefinitely. (3) attempts to pg_cancel_backend on the DROP DATABASE process were futile. yet no other queries were accessing that database according to pg_stat_activity. (4) pg_ctl -m fast did not shut down the server. (5) pg_ctl -m immediate did shut it down. (5a) realized pgbouncer was still running. shut it down as well. (6) on restart, the server took several minutes to *start* recover mode. then it got to a certain log segment, and hung indefinitely. (7) killed postgres (8) restarted again. this time the server came up in about 20s, and is behaving normally. The deleted database is not there. Setup: RHEL 5 PostgreSQL 8.4.2, installed from yum.postgresql.org 8 databases and about 10G of data on the server pgbouncer running but NOT connected to the dropped database, just to two others I've made a full copy of my PGDATA directory if anyone wants me to test anything.
On Fri, Feb 5, 2010 at 9:50 PM, The Fuzzy Chef <fuzzy@fuzzychef.org> wrote: > Just had a very odd thing happen with an 8.4.2 database, and am > concerned it's an undiagnosed bug: > > (1) created and dropped the same database 10-15 times in the course of > running some tests. > > (2) at one point, dropdb hung. =A0Indefinitely. > > (3) attempts to pg_cancel_backend on the DROP DATABASE process were > futile. =A0yet no other queries were accessing that database according to > pg_stat_activity. > > (4) pg_ctl -m fast did not shut down the server. > > (5) pg_ctl -m immediate did shut it down. > (5a) realized pgbouncer was still running. =A0shut it down as well. > > (6) on restart, the server took several minutes to *start* recover mode. > =A0then it got to a certain log segment, and hung indefinitely. > > (7) killed postgres > > (8) restarted again. =A0this time the server came up in about 20s, and is > behaving normally. =A0The deleted database is not there. > > Setup: > RHEL 5 > PostgreSQL 8.4.2, installed from yum.postgresql.org > 8 databases and about 10G of data on the server > pgbouncer running but NOT connected to the dropped database, just to two > others > > I've made a full copy of my PGDATA directory if anyone wants me to test > anything. It's too bad you didn't capture a stack backtrace at step #3 or step #6. If you manage to reproduce the problem, that would be a good thing to try to get. ...Robert
> It's too bad you didn't capture a stack backtrace at step #3 or step > #6. If you manage to reproduce the problem, that would be a good > thing to try to get. It never actually crashed. And, of course, this was happening right when I needed to go home and the server needed to be up for that. --Josh
On Sat, Feb 6, 2010 at 7:43 PM, The Fuzzy Chef <fuzzy@fuzzychef.org> wrote: > >> It's too bad you didn't capture a stack backtrace at step #3 or step >> #6. =A0If you manage to reproduce the problem, that would be a good >> thing to try to get. > > Well, I never got an actual crash. That's OK - you can still attach gdb and see where it's hung up... it would have been really nice to see what that "hung" drop database thought it was doing... ...Robert
On 2/6/10 8:20 PM, Robert Haas wrote: > On Sat, Feb 6, 2010 at 7:43 PM, The Fuzzy Chef <fuzzy@fuzzychef.org> wrote: >>> It's too bad you didn't capture a stack backtrace at step #3 or step >>> #6. If you manage to reproduce the problem, that would be a good >>> thing to try to get. >> Well, I never got an actual crash. > > That's OK - you can still attach gdb and see where it's hung up... it > would have been really nice to see what that "hung" drop database > thought it was doing... Yep, sorry. Was trying to get the system working first, and then afterwards thought it might be worth reporting. Anything I can mine out of the logs or files? --Josh Berkus
On Sun, Feb 7, 2010 at 12:00 AM, Josh Berkus <josh@agliodbs.com> wrote: > On 2/6/10 8:20 PM, Robert Haas wrote: >> On Sat, Feb 6, 2010 at 7:43 PM, The Fuzzy Chef <fuzzy@fuzzychef.org> wro= te: >>>> It's too bad you didn't capture a stack backtrace at step #3 or step >>>> #6. =A0If you manage to reproduce the problem, that would be a good >>>> thing to try to get. >>> Well, I never got an actual crash. >> >> That's OK - you can still attach gdb and see where it's hung up... =A0it >> would have been really nice to see what that "hung" drop database >> thought it was doing... > > Yep, sorry. =A0Was trying to get the system working first, and then > afterwards thought it might be worth reporting. > > Anything I can mine out of the logs or files? I'm not familiar enough to say; but I wonder if you could try to reproduce the problem on (a copy of) your saved directory. ...Robert