Re: bg worker: overview

Hey Dimitri,

On 07/24/2010 07:26 PM, Dimitri Fontaine wrote:
> Trying to figure out how it would fit the PGQ and pgagent needs. But
> maybe user defined daemons should be sub-coordinators (I used to think
> about them as "supervisors") able to talk to the coordinator to get a
> backend connected to some given database and distribute work to it.

Hm.. sounds like an awful lot of work to me, but if you need the 
separation and security of a separate process...

To simplify, you might want to start a bgworker on database 'postgres', 
which then acts as a sub-coordinator (and doesn't really need to use its 
database connection).

> You're using iMessage as the data exchange, how are you doing the work
> distribution? What do you use to tell the backend what is the processing
> you're interrested into?

Well, there are different types of imessages defined in imsg.h. If you 
are coding something within Postgres, you'd just add all the required 
messages types there. There's no such thing as an external registration 
for new message types.

For example, for autovacuum, there are two message types: 
IMSGT_PERFORM_VACUUM, that's sent from the coordinator to a bgworker, 
and initiates a vacuum job there. Then there's IMSGT_FORCE_VACUUM, which 
is sent from a backend to the coordinator to inform it that a certain 
database urgently needs vacuuming.

For Postgres-R, things are a bit more complicated. The first IMSGT_CSET 
messages starts the application of a remote transaction. Further 
IMSGT_CSET messages may follow. The IMSGT_ORDERING message finally 
completes the job.

So, imessage types cannot be mapped to jobs directly. See 
include/postmaster/coordinator.h, enum worker_state. Those are the 
possible states a worker can be in (job types).

Adding a job would consist of adding a worker_state, plus at least one 
imessage type. Once the worker is done with its job, it returns 
IMSGT_READY to the coordinator.

I'm open to refinements, such as assigning a certain range of message 
types to external use or some such. However, have no idea how to avoid 
clashing message type ids, then. Maybe those should still be part of imsg.h?

>> Go ahead, re-use the background workers. That's what I've published
>> them for
>
> Hehe :) The aim of this thread would be to have your input as far as
> designing an API would go, now that we're about on track as to what the
> aim is.

Oh, sure. :-)

> In very short, the idea is a clock that ticks and associate
> current_txid() to now(), so that you're able to say "give me 3s worth of
> transactions activity from this queue". It then provides facilities to
> organise a queue into batches at consumer request, and for more details,
> see there:
>
>    http://github.com/markokr/skytools-dev/blob/master/sql/ticker/pgqd.c
>    http://github.com/markokr/skytools-dev/blob/master/sql/ticker/ticker.c

Okay, thanks for the pointers. However, comments are relatively sparse 
in there as well...

> But the important thing as far as making it a child of the coordinator
> goes would be, I guess, that it's some C code running as a deamon and
> running SQL queries from time to time. The SQL queries are calling C
> user defined functions, provided by the PGQ backend module.

You could certainly define jobs, which don't ever terminate. And calling 
SQL queries certainly sounds more like a background job to me, than 
something belonging to the sphere of the coordinator. Sorry, if my first 
impulse has been misleading.

So, the bgworker infrastructure could probably satisfy the internal 
communication needs. But how does this ticker daemon talk to the 
outside? Does it need to open a socket and listen there? Or do the 
requests to that queue come in via SQL?

Regards

Markus