Re: parallel restore

I wrote:
>
>
> Tom Lane wrote:
>> Andrew Dunstan <andrew@dunslane.net> writes:
>>  
>>> Tom Lane wrote:
>>>    
>>>> There is an unfinished TODO item here: we really ought to make it work
>>>> for tar-format archives.  That's probably not hugely difficult, but
>>>> I didn't look into it, and don't think we should hold up applying the
>>>> existing patch for it.
>>>>       
>>
>>  
>>> Right. Were you thinking this should be done for 8.4?
>>>     
>>
>> If you have time to look into it, sure.  Otherwise we should just put it
>> on the TODO list.
>>
>>            
>>   
>
> I've had a look at this. If our tar code supported out of order 
> restoration(using fseeko) I'd be done. But it doesn't, and I won't get 
> that done for 8.4, if at all. I'm not sure what would be involved in 
> making it work.
>
>

OK, I've spent some more time on this. pg_dump when writing a custom 
format file writes out the header and table of contents and then the 
data members, keeping track of where each one starts. If the output is 
seekable (as it usually is) it then rewrites the table of contents, this 
time including the data member offsets. Parallel restore requires that 
this offset info be available, and if the pg_dump output file was not 
seekable by pg_dump (e.g. if it was a pipe) then it will be unsuitable 
for use with parallel restore, which will fail.

In the case of tar output, pg_dump doesn't make any effort to keep the 
offset info at all, so parallel restore is not currently suitable for 
use with tar output, regardless of whether or not the pg_dump output was 
seekable.

I think we could cure both of these cases by having pg_dump write out a 
second copy of the table of contents, including data member offsets, at 
the end of the archive. Or it might just be a table of <data-member-id, 
offset> pairs if we're worried about space. In the latter case we'd need 
to go back and fix up the TOC, but that would be fairly simple. Either 
way I think we'd need to bump the archive version number so we'd know 
when to expect this.

Once we have that the custom format code should fail on this no matter 
how the dump was made, and parallel restore should work with tar format 
once we add code to it to seek for data members.

I think all of this can wait to 8.5, except that we should possibly 
document a bit more completely the current limitations on parallel restore.

(I was initially tempted to say we'd need compression of individual data 
members in tar format to do this sanely, but since the 
offsets-at-the-end suggestion should work even when pg_dump is 
outputting to a pipe, we'd still be able to send the output through gzip 
and so get a conventional .tgz file.)

cheers

andrew