Re: errors on restoring postgresql binary dump to glusterfs

Thank you Magnus for all the inputs. If I get any comments from
gluster community, I will update here.

Liang

On Mon, May 7, 2012 at 3:27 PM, Magnus Hagander <magnus@hagander.net> wrote:
> On Mon, May 7, 2012 at 7:34 PM, Liang Ma <ma.satops@gmail.com> wrote:
>> On Mon, May 7, 2012 at 12:54 PM, Magnus Hagander <magnus@hagander.net> wrote:
>>> On Mon, May 7, 2012 at 5:02 PM, Liang Ma <ma.satops@gmail.com> wrote:
>>>> On Fri, May 4, 2012 at 3:58 AM, Magnus Hagander <magnus@hagander.net> wrote:
>>>>> On Mon, Apr 30, 2012 at 8:34 PM, Liang Ma <ma.satops@gmail.com> wrote:
>>>>>> Hi There,
>>>>>>
>>>>>> While trying to restore a ~700GM binary dump by command
>>>>>>
>>>>>> pg_restore -d dbdata < sampledbdata-20120327.pgdump
>>>>>>
>>>>>> I encountered following errors repeatedly
>>>>>>
>>>>>> pg_restore: [archiver (db)] Error from TOC entry 2882463; 2613
>>>>>> 10267347 BLOB 10267347 sdmcleod
>>>>>> pg_restore: [archiver (db)] could not execute query: ERROR:
>>>>>> unexpected data beyond EOF in block 500 of relation base/16386/11743
>>>>>> HINT:  This has been seen to occur with buggy kernels; consider
>>>>>> updating your system.
>>>>>
>>>>> Note the message right here...
>>>>>
>>>>> There may be further indications in the server log about what's wrong.
>>>>>
>>>>
>>>> The server's logs in message file were clean.
>>>
>>> Then your logging is incorrectly configured, because it should *at
>>> least* have the same message as the one that showed up in the client.
>>>
>>
>> Oh, yes, the same error messages were logged in the postgresql log
>> file but no further information. I thought you implied that there may
>> be some indication in server's system logs, which I couldn't find any.
>
> Well, there might be, I wasn't sure :-) I guess there wasn't.
>
>
>>>>>> The server runs Ubuntu server 10.04 LTS with postgresql upgraded to
>>>>>> version 9.1.3-1~lucid. The postgresql data directory is located in a
>>>>>> glusterfs mounted directory to a replicated volume vol-2
>>>>>
>>>>> I assume you don't have more than one node actually *accessing* the
>>>>> data directory at the same time, right?
>>>>>
>>>>
>>>> Yes, you are right. I just set up this glusterfs and postgresql server
>>>> with two nodes for testing purpose. There was no other gluster
>>>> filesystem access activity at the time I tried to restore the
>>>> postgresql dump. Do you know if postgresql recommends any other
>>>> cluster filesystem, or it may not like cluster filesystem at all?
>>>
>>>
>>> Did you have PostgreSQL started on both nodes? That is *not*
>>> supported. If PostgreSQL only runs on one node at a time it should in
>>> theory work, provided the cluster filesystem provides all the services
>>> that a normal filesystem does, such as respecting fsync.
>>>
>>
>> Postgresql are installed in both nodes, but only one node's postgresql
>> data directory points to glusterfs filesystem. Another one's data
>> directory is in its default location in the local ext4 filesystem.
>> This is the one I used to prove the dump file can be restored without
>> any problem when glusterfs is not involved.
>
> ok. That should in theory be safe. Having two active notes against th
> efilesystem is never safe.
>
>
>> According to its introduction and document, glusterfs is supposed to
>> appear as a normal filesystem when being mounted, although I don't
>> know how well it respects things like fsync.
>
> It certainly looks like it's failing at some point. So yeah, I'm
> pretty sure you need to get in touch with the glusterfs folks -
> hopefully you get a response from them soon.
>
> --
>  Magnus Hagander
>  Me: http://www.hagander.net/
>  Work: http://www.redpill-linpro.com/