Re: background processes vs. hot standby

On Fri, May 24, 2013 at 12:35 PM, Robert Haas <robertmhaas@gmail.com> wrote:
> But I don't know what to do about the problem of needing to know how
> many backends there are.  I agree with Andres that it's not very
> friendly to enforce a restriction that all the same worker processes
> must be present on the standby.

OK, I have an idea for fixing this.  This would be 9.4 material; I'm
happy enough to leave this broken for 9.3, as the chances of anyone
hitting it in practice seem remote, and if they do, the workaround is
not hard.

What I'd like to do is change the way that we allocate slots for
bgworkers.  Instead of computing the number of slots based on how many
bgworkers are registered at postmaster startup time, I'd like to
allocate a fixed number of slots, controlled by a GUC called
max_worker_processes, which will obviously be PGC_POSTMASTER.  I would
propose to give this GUC a small but non-zero default value; I was
thinking of 8.

Under this proposal, it would no longer possible to register an
unlimited number of background workers without any changes to
postgresql.conf; if you want to register more of them than the default
value of this parameter, you'll need to increase the parameter.
That's a downside, but it's one that I think will affect very few
users; how many people are going to be running installations with more
than 8 background workers?  (And, those that are probably pretty
sophisticated users who won't suffer greatly from having to change two
GUCs rather than just one.)

So what's the upside?  One advantage, obviously, is that we fix this
bug.  max_worker_processes would get propagated the standby, and
checked, just as max_connections does today.  The standby doesn't have
to actually load the same extensions; it just has to allow for at
least as many slots, which is cleaner.

But that's pretty minor.  I see a much larger advantage: we wouldn't
be limited to worker procesess that are started up at fixed times in
the postmaster lifecycle, as we currently are.  Right now, background
workers can only be started at three points:
BgWorkerStart_PostmasterStart, BgWorkerStart_ConsistentState, and
BgWorkerStart_RecoveryFinished.  This change would allow us to move
towards worker processes that can be started "on the fly", whenever
needed, just as long as there are available slots from the total
allocation carved out by max_worker_processes.

I think you can see where I'm going here: aside from giving extension
code more options in terms of how it uses worker processes, this would
give us a facility that would be a useful building block toward
parallel sort/copy/query/whatever.  Even without parallelism in core,
it's not too difficult to imagine a contrib module that does some kind
of ad-hoc parallelism by firing up worker backends, which would be
awkward with the 9.3 infrastructure.  But the real big win is that the
core facilities, when and as they arrive, can draw from that same pool
of workers.

Thoughts?

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company