Re: dynamic background workers

Поиск
Список
Период
Сортировка
От Michael Paquier
Тема Re: dynamic background workers
Дата
Msg-id CAB7nPqR8pySQFcsrfVrC2fHwWtaLQQ4UMzHyH5UM594pKd4kxA@mail.gmail.com
обсуждение исходный текст
Ответ на dynamic background workers  (Robert Haas <robertmhaas@gmail.com>)
Список pgsql-hackers
On Sat, Jun 15, 2013 at 6:00 AM, Robert Haas <robertmhaas@gmail.com> wrote:
The second patch, dynamic-bgworkers-v1.patch, revises the background
worker API to allow background workers to be started dynamically.
This requires some communication channel from ordinary workers to the
postmaster, because it is the postmaster that must ultimately start
the newly-registered workers.  However, that communication channel has
to be designed pretty carefully, lest a shared memory corruption take
out the postmaster and lead to inadvertent failure to restart after a
crash.  Here's how I implemented that: there's an array in shared
memory of a size equal to max_worker_processes.  This array is
separate from the backend-private list of workers maintained by the
postmaster, but the two are kept in sync.  When a new background
worker registration is added to the shared data structure, the backend
adding it uses the existing pmsignal mechanism to kick the postmaster,
which then scans the array for new registrations.  I have attempted to
make the code that transfers the shared_memory state into the
postmaster's private state as paranoid as humanly possible.  The
precautions taken are documented in the comments.  Conversely, when a
background worker flagged as BGW_NEVER_RESTART is considered for
restart (and we decide against it), the corresponding slot in the
shared memory array is marked as no longer in use, allowing it to be
reused for a new registration.

Since the postmaster cannot take locks, synchronization between the
postmaster and other backends using the shared memory segment has to
be lockless.  This mechanism is also documented in the comments.  An
lwlock is used to prevent two backends that are both registering a new
worker at about the same time from stomping on each other, but the
postmaster need not care about that lwlock.

This patch also extends worker_spi as a demonstration of the new
interface.  With this patch, you can CREATE EXTENSION worker_spi and
then call worker_spi_launch(int4) to launch a new background worker,
or combine it with generate_series() to launch a bunch at once.  Then
you can kill them off with pg_terminate_backend() and start some new
ones.  That, in my humble opinion, is pretty cool.
This looks really interesting, +1. I'll test the patch if possible next week.
--
Michael

В списке pgsql-hackers по дате отправления:

Предыдущее
От: Andres Freund
Дата:
Сообщение: Re: pluggable compression support
Следующее
От: Robert Haas
Дата:
Сообщение: Re: pluggable compression support