Обсуждение: ERROR: unexpected data beyond EOF
Alright folks, So I have this error: postgres[21118]: [8-1] ERROR: unexpected data beyond EOF in block 9 of relation base/430666195/430666206 Which produces this lovely hint: postgres[21118]: [8-2] HINT: This has been seen to occur with buggy kernels; consider updating your system. However, the problem is, all relevant information we can find is Linux/NFS based. Now it is no secret that Pg does some weird things on NFS or Virtualized volumes but I am not sure where to even start with this. The system is: FreeBSD 10 ZFS iSCSI RAID 50 (don't start, I didn't spec it). fsync on, full_page_writes on The restart of PostgreSQL makes the error go away and things progress normally. We don't experience further errors etc... Any thoughts on this? JD -- Command Prompt, Inc. - http://www.commandprompt.com/ 503-667-4564 PostgreSQL Centered full stack support, consulting and development. Announcing "I'm offended" is basically telling the world you can't control your own emotions, so everyone else should do it for you.
Joshua D. Drake wrote: > > Alright folks, > > So I have this error: > > postgres[21118]: [8-1] ERROR: unexpected data beyond EOF > in block 9 of relation base/430666195/430666206 > > Which produces this lovely hint: > > postgres[21118]: [8-2] HINT: This has been seen to occur with buggy > kernels; consider updating your system. > > However, the problem is, all relevant information we can find is Linux/NFS > based. I have vague recollections of seeing this relatively recently on tables that were truncated often. Is that the case here? -- Álvaro Herrera http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
On 04/30/2015 10:28 AM, Alvaro Herrera wrote: > > Joshua D. Drake wrote: >> >> Alright folks, >> >> So I have this error: >> >> postgres[21118]: [8-1] ERROR: unexpected data beyond EOF >> in block 9 of relation base/430666195/430666206 >> >> Which produces this lovely hint: >> >> postgres[21118]: [8-2] HINT: This has been seen to occur with buggy >> kernels; consider updating your system. >> >> However, the problem is, all relevant information we can find is Linux/NFS >> based. > > I have vague recollections of seeing this relatively recently on tables > that were truncated often. Is that the case here? Just dug into the table a bit and yes, it appears to be a transform table. Further questions/thoughts? JD -- Command Prompt, Inc. - http://www.commandprompt.com/ 503-667-4564 PostgreSQL Centered full stack support, consulting and development. Announcing "I'm offended" is basically telling the world you can't control your own emotions, so everyone else should do it for you.
I take that back, it appears this table is heavily deleted from and also uses the lo_manage() triggers. -- Command Prompt, Inc. - http://www.commandprompt.com/ 503-667-4564 PostgreSQL Centered full stack support, consulting and development. Announcing "I'm offended" is basically telling the world you can't control your own emotions, so everyone else should do it for you.
Joshua D. Drake wrote: > > I take that back, it appears this table is heavily deleted from and also > uses the lo_manage() triggers. Well, if it's heavily deleted, then it's probably also heavily vacuumed and from time to time empty pages at the tail are removed by vacuum. It might also be the case I was remembering, rather than regular TRUNCATE. I don't think the vacuumlo stuff would have much to do with it issue; I think it would only scan the table, then delete stuff from pg_largeobject. It doesn't modify the table itself (unless I'm misremembering) Anyway, I don't remember that we reached any useful conclusion. Andres suspected a PG bug, but we didn't find anything. -- Álvaro Herrera http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
On 04/30/2015 12:09 PM, Alvaro Herrera wrote: > > Joshua D. Drake wrote: >> >> I take that back, it appears this table is heavily deleted from and also >> uses the lo_manage() triggers. > > Well, if it's heavily deleted, then it's probably also heavily vacuumed > and from time to time empty pages at the tail are removed by vacuum. It > might also be the case I was remembering, rather than regular TRUNCATE. > > I don't think the vacuumlo stuff would have much to do with it issue; I > think it would only scan the table, then delete stuff from > pg_largeobject. It doesn't modify the table itself (unless I'm > misremembering) > > Anyway, I don't remember that we reached any useful conclusion. Andres > suspected a PG bug, but we didn't find anything. Well it certainly causes an outage unfortunately. Once the error occurs postgresql will stop accepting requests (which is correct but still). JD -- Command Prompt, Inc. - http://www.commandprompt.com/ 503-667-4564 PostgreSQL Centered full stack support, consulting and development. Announcing "I'm offended" is basically telling the world you can't control your own emotions, so everyone else should do it for you.
Hi, On 2015-04-30 10:05:33 -0700, Joshua D. Drake wrote: > postgres[21118]: [8-1] ERROR: unexpected data beyond EOF > in block 9 of relation base/430666195/430666206 > > FreeBSD 10 > ZFS > iSCSI > RAID 50 (don't start, I didn't spec it). > fsync on, full_page_writes on > > The restart of PostgreSQL makes the error go away and things progress > normally. We don't experience further errors etc... A couple questions: 1) Do you know what 430666206 refers to? Something like SELECT oid::regclass, relkind FROM pg_class WHERE pg_relation_filenode(430666206) in the correct database should answer. 2) Any chance you have zero_damaged_pages enabled? 3) Does the issue still occur? Is there any chance of manual investigation before restaring the database? Andres