Обсуждение: Unexpected data beyond EOF during heavy writes
Hello everyone,
We are seeing the following error message occasionally in the postgres logs:
2010-05-13 23:49:03 PDT ERROR: unexpected data beyond EOF in block 4106698 of relation "custom_discoveryprofile"
2010-05-13 23:49:03 PDT HINT: This has been seen to occur with buggy kernels; consider updating your system.
I have read several threads on this, including:
http://archives.postgresql.org/pgsql-general/2007-03/msg01535.php
and
http://archives.postgresql.org/pgsql-general/2009-07/msg01011.php
I am wondering if anyone has ever placed a bug report to any of the linux vendors or open source sites?
I could not find a bug report that matched this at http://bugzilla.redhat.com/ and am trying to track down what is needed to resolve the error.
We are running a an IBM Blade Center server 64 bit Red Hat 5.3 kernel with 16 cores doing heavy I/O to the database.
uname -a
Linux 2.6.18-128.el5 #1 SMP Wed Dec 17 11:41:38 EST 2008 x86_64 x86_64 x86_64 GNU/Linux x86_64 x86_64 x86_64 GNU/Linux
Thanks for any pointers,
Tony Sullivan
Excerpts from Tony Sullivan's message of mar may 18 13:19:13 -0400 2010: > Hello everyone, > > We are seeing the following error message occasionally in the postgres logs: > > 2010-05-13 23:49:03 PDT ERROR: unexpected data beyond EOF in block 4106698 of relation "custom_discoveryprofile" > 2010-05-13 23:49:03 PDT HINT: This has been seen to occur with buggy kernels; consider updating your system. What's your storage? --
<div class="Section1"><pre><font face="Courier New" size="2"><span style="font-size:10.0pt">> Hello everyone,</span></font></pre><pre><fontface="Courier New" size="2"><span style="font-size:10.0pt">> </span></font></pre><pre><fontface="Courier New" size="2"><span style="font-size:10.0pt">> We are seeing the followingerror message occasionally in the postgres logs:</span></font></pre><pre><font face="Courier New" size="2"><spanstyle="font-size:10.0pt">> </span></font></pre><pre><font face="Courier New" size="2"><span style="font-size:10.0pt">>2010-05-13 23:49:03 PDT ERROR: unexpected data beyond EOF in block 4106698 of relation "custom_discoveryprofile"</span></font></pre><pre><fontface="Courier New" size="2"><span style="font-size:10.0pt">> 2010-05-1323:49:03 PDT HINT: This has been seen to occur with buggy kernels; consider updating your system.</span></font></pre><pre><fontface="Courier New" size="2"><span style="font-size:10.0pt"> </span></font></pre><pre><fontface="Courier New" size="2"><span style="font-size:10.0pt">What'syour storage?</span></font></pre><pre><font face="Courier New" size="2"><span style="font-size:10.0pt"> </span></font></pre><pre><fontface="Courier New" size="2"><span style="font-size:10.0pt">-- </span></font></pre><pre><fontface="Courier New" size="2"><span style="font-size:10.0pt"> </span></font></pre><p class="MsoNormal"style="text-autospace:none"><font face="Arial" size="2"><span style="font-size:10.0pt;font-family:Arial">Itis NetApp storage - </span></font><font face="Courier New" size="2"><span style="font-size:10.0pt;font-family:"CourierNew"">a FAS3070 running Data ONTAP 7.3.2</span></font><p class="MsoNormal" style="text-autospace:none"><fontface="Courier New" size="2"><span style="font-size:10.0pt;font-family:"Courier New""> </span></font><pclass="MsoNormal" style="text-autospace:none"><font face="Courier New" size="2"><span style="font-size:10.0pt;font-family:"CourierNew"">Here are the mount options</span></font><p class="MsoNormal" style="text-autospace:none"><fontface="Courier New" size="2"><span style="font-size:10.0pt;font-family:"Courier New""> </span></font><pclass="MsoNormal" style="text-autospace:none"><font face="Courier New" size="2"><span style="font-size:10.0pt;font-family:"CourierNew""><server>:/vol/sw on /x/eng/sw type nfs</span></font><p class="MsoNormal"style="text-autospace:none"><font face="Courier New" size="2"><span style="font-size:10.0pt;font-family:"Courier New"">(rw,intr,hard,rsize=32768,wsize=32768,nfsvers=3,timeo=600,tcp,nointr,addr=xx.xx.</span></font><pclass="MsoNormal" style="text-autospace:none"><fontface="Courier New" size="2"><span style="font-size:10.0pt;font-family:"Courier New"">xx.xx)</span></font><pclass="MsoNormal" style="text-autospace:none"><font face="Courier New" size="2"><span style="font-size:10.0pt;font-family:"CourierNew""> </span></font><p class="MsoNormal"><font face="Arial" size="2"><span style="font-size:10.0pt; font-family:Arial">Thanks,</span></font><p class="MsoNormal"><font face="Arial" size="2"><span style="font-size:10.0pt; font-family:Arial"> </span></font><p class="MsoNormal"><strong><b><font face="Arial" size="2"><span style="font-size:10.0pt;font-family:Arial">TonySullivan</span></font></b></strong> <strong><b><font face="Arial" size="2"><spanstyle="font-size:10.0pt;font-family: Arial"></span></font></b></strong><p class="MsoNormal"><font face="Times New Roman" size="3"><span style="font-size: 12.0pt"> </span></font></div>
Excerpts from Tony Sullivan's message of jue may 20 16:54:17 -0400 2010: > > Hello everyone, > > > > > > We are seeing the following error message occasionally in the postgres logs: > > > > > > 2010-05-13 23:49:03 PDT ERROR: unexpected data beyond EOF in block 4106698 of relation "custom_discoveryprofile" > > > 2010-05-13 23:49:03 PDT HINT: This has been seen to occur with buggy kernels; consider updating your system. > > What's your storage? This was added here http://archives.postgresql.org/message-id/20060925220110.76B6A9FB251@postgresql.org in response to these two: http://thread.gmane.org/gmane.comp.db.postgresql.admin/18807 http://thread.gmane.org/gmane.comp.db.postgresql.general/74532 We (at Command Prompt) researched this recently for another setup and the common point you both have is NetApp. I then wondered about a bug in NetApp driver or NFS client implementation. --
On Thu, May 20, 2010 at 3:19 PM, Alvaro Herrera <alvherre@alvh.no-ip.org> wrote: > We (at Command Prompt) researched this recently for another setup and > the common point you both have is NetApp. I then wondered about a bug > in NetApp driver or NFS client implementation. It's definitely not (just) NetApp, though it may be their NFS -- or NFS in general; I couldn't say. I can't speak to their NFS implementation, beyond having generally heard good things about it, but I've run PostgreSQL on filers for years, and have never seen that message. Granted, I've only been iSCSI- or fibre-attached (or had the storage path abstracted away by some form of virtualization), so I haven't seen every possible use-case. In general, though, I'd be pretty wary of running postgres on an NFS mount. I know a lot of people run Oracle that way, but at the filesystem level, there are some vast differences between the two. Has anyone ever seen this message on non-NetApp NFS? rls -- :wq
Rosser Schwarz <rosser.schwarz@gmail.com> writes: > Has anyone ever seen this message on non-NetApp NFS? It's been seen on non-NFS storage: http://archives.postgresql.org/pgsql-admin/2006-09/msg00096.php I don't believe we implicated NFS in the other original report, either. However, it's certainly possible that there's a similar bug in the NFS stack too on some platforms. regards, tom lane