Re: Backends stalled in 'startup' state: index corruption

Greg Sabino Mullane <greg@endpoint.com> writes:

Yeah, this is proof that what it was doing is the same as what we saw in

Jeff's backtrace, ie loading up the system catalog relcache entries the

hard way via seqscans on the core catalogs.  So the question to be

answered is why that's suddenly a big performance bottleneck.  It's not

a cheap operation of course (that's why we cache the results ;-)) but

it shouldn't take minutes either.  And, because they are seqscans, it

doesn't seem like messed-up indexes should matter.

FWIW, this appeared to be an all-or-nothing event: either every new backend

was suffering through this, or none were. They all seemed to clear up

at the same time as well.

Mostly not surprising.  They'd definitely all hit the missing init file
at the same time, so the stalling would start consistently for all.  And
once any one process successfully created a new file, subsequent incoming
sessions wouldn't stall.  However, the remaining processes trying to
compute new init files would still have to complete the process, so I'd
expect there to be a diminishing effect --- the ones that were stalling
shouldn't all release exactly together.  Unless there is some additional
effect that's syncing them all.  (I wonder for instance if the syncscan
logic is kicking in here.)

1. Somebody decides to update one of those rows, and it gets dropped in
some remote region of the table.  The only really plausible reason for
this is deciding to fool with the column-specific stats target
(attstattarget) of a system catalog.  Does that sound like something
either of you might have done?  You could check it by looking at the
ctid columns of the pg_attribute rows for system catalogs, and seeing
if any have large block numbers.