Re: parallel pg_restore

On Mon, 2008-09-22 at 11:38 -0400, Andrew Dunstan wrote:
> 
> Simon Riggs wrote:
> > On Mon, 2008-09-22 at 09:53 +0200, Dimitri Fontaine wrote:
> >
> >   
> >>> My intention is to have single-thread restore remain the default, at
> >>> least for this go round, and have the user be able to choose
> >>> --multi-thread=nn to specify the number of concurrent connections to use.
> >>>       
> >> What about the make famous -j option?
> >>
> >>        -j [jobs], --jobs[=jobs]
> >>             Specifies the number of jobs (commands) to run simultaneously.  If
> >>             there  is  more than one -j option, the last one is effective.  If
> >>             the -j option is given without an argument, make  will  not  limit
> >>             the number of jobs that can run simultaneously.
> >>     
> >
> > +1
> >
> >   
> 
> If that's the preferred name I have no problem. I'm not sure about the 
> default argument part, though.
> 
> First, I'm not sure out getopt infrastructure actually provides for 
> optional arguments, and I am not going to remove it in pg_restore to get 
> around such a problem, at least now.
> 
> More importantly, I'm not convinced it's a good idea. It seems more like 
> a footgun that will potentially try to launch thousands of simultaneous 
> restore connections. I should have thought that optimal performance 
> would be reached at some small multiple (say maybe 2?) of the number of 
> CPUs on the server. You could achieve unlimited parallelism by saying 
> something like --jobs=99999, but I'd rather that were done very 
> explicitly instead of as the default value of the parameter.

OK, sounds best.

-- Simon Riggs           www.2ndQuadrant.comPostgreSQL Training, Services and Support