Re: parallel pg_restore

On Sun, 2008-09-21 at 18:15 -0400, Tom Lane wrote:
> Andrew Dunstan <andrew@dunslane.net> writes:
> > I am working on getting parallel pg_restore working. I'm currently 
> > getting all the scaffolding working, and hope to have a naive prototype 
> > posted within about a week.
> 
> > The major question is how to choose the restoration order so as to 
> > maximize efficiency both on the server and in reading the archive.
> 
> One of the first software design principles I ever learned was to
> separate policy from mechanism.  ISTM in this first cut you ought to
> concentrate on mechanism and let the policy just be something dumb
> (but coded separately from the infrastructure).  We can refine it after
> that.

Agreed. We musn't make too many built in assumptions about the best way
to parallelise the restore.

For example, running all CREATE INDEX at same time may help I/O on the
scan but it may also swamp memory and force additional I/O as a result.

We might need a setting for total memory available, so pg_restore can
try not to run tasks that will exceed that across settings. Preferably
this wouldn't be just a pg_restore setting.

-- Simon Riggs           www.2ndQuadrant.comPostgreSQL Training, Services and Support