Re: buildfarm: could not read block 3 in file "base/16384/2662":read only 0 of 8192 bytes

Hi,

On 2018-07-20 13:24:50 -0700, Andres Freund wrote:
> On 2018-07-20 16:15:14 -0400, Tom Lane wrote:
> > We've seen several occurrences of $subject in the buildfarm in the past
> > month or so.  Scraping the logs, I find
> > 
> >  coypu    | 2018-06-14 21:17:49 | HEAD          | Check | 2018-06-14 23:31:44.505 CEST [5b22deb8.30e1:124] ERROR:
couldnot read block 3 in file "base/16384/2662": read only 0 of 8192 bytes
 
> >  coypu    | 2018-06-14 21:17:49 | HEAD          | Check | 2018-06-14 23:31:44.653 CEST [5b22deb8.6d4d:132] ERROR:
couldnot read block 3 in file "base/16384/2662": read only 0 of 8192 bytes
 
> >  gull     | 2018-06-21 04:02:11 | HEAD          | Check | 2018-06-20 21:27:06.843 PDT [31138:3] FATAL:  could not
readblock 3 in file "base/16384/2662": read only 0 of 8192 bytes
 
> >  mandrill | 2018-06-22 16:14:16 | HEAD          | Check | 2018-06-22 16:46:24.138 UTC [14353240:43]
pg_regress/create_table_likeERROR:  could not read block 3 in file "base/16384/2662": read only 0 of 8192 bytes
 
> >  mandrill | 2018-06-22 16:14:16 | HEAD          | Check | 2018-06-22 16:46:24.137 UTC [20710034:1] ERROR:  could
notread block 3 in file "base/16384/2662": read only 0 of 8192 bytes
 
> >  lapwing  | 2018-07-05 07:20:02 | HEAD          | Check | 2018-07-05 07:21:45.585 UTC [24022:1] ERROR:  could not
readblock 3 in file "base/16384/2662": read only 0 of 8192 bytes
 
> >  lapwing  | 2018-07-05 07:20:02 | HEAD          | Check | 2018-07-05 07:21:45.591 UTC [23941:39]
pg_regress/roleattributesERROR:  could not read block 3 in file "base/16384/2662": read only 0 of 8192 bytes
 
> >  mantid   | 2018-07-17 01:20:04 | REL_11_STABLE | Check | 2018-07-16 21:21:32.557 EDT [26072:1] ERROR:  could not
readblock 3 in file "base/16384/2662": read only 0 of 8192 bytes
 
> >  mantid   | 2018-07-17 01:20:04 | REL_11_STABLE | Check | 2018-07-16 21:21:32.557 EDT [25990:82] ERROR:  could not
readblock 3 in file "base/16384/2662": read only 0 of 8192 bytes
 
> >  coypu    | 2018-07-17 01:27:39 | HEAD          | Check | 2018-07-17 03:47:26.264 CEST [5b4d4aa4.3ac4:71] ERROR:
couldnot read block 3 in file "base/16384/2662": read only 0 of 8192 bytes
 
> >  coypu    | 2018-07-20 11:18:02 | HEAD          | Check | 2018-07-20 13:32:14.104 CEST [5b51c833.4884:131] ERROR:
couldnot read block 3 in file "base/16384/2662": read only 0 of 8192 bytes
 
> > 
> > While I didn't go back indefinitely far, there are no other occurrences of
> > "could not read block" failures in the last three months.  This suggests
> > strongly that we broke something in early June.  Don't know what though.
> > Ideas?
> > 
> > In case anyone's wondering, 2662 is pg_class_oid_index.
> 
> While I do not immediately see how, and the affected versions don't
> really point that way, the timing does make me think of
> http://git.postgresql.org/gitweb/?p=postgresql.git;a=commit;h=a54e1f1587793b5bf926630ec9ce142e4578d7a0
> 
> HEAD/REL_11_STABLE apparently solely being affected points elsewhere,
> but I don't immediatley know where.

Hm, there was:
http://archives.postgresql.org/message-id/20180628150209.n2qch5jtn3vt2xaa%40alap3.anarazel.de

I don't immediately see it being responsible, but I wonder if there's a
chance it actually is: Note that it happens in a parallel group that
includes vacuum.sql, which does a VACUUM FULL pg_class - but I still
don't immediately see how it could apply.

But either way, we need to fix the issue mentioned in the above email.

Greetings,

Andres Freund