Re: relaying errors from background workers

On Thu, May 22, 2014 at 6:33 AM, Petr Jelinek <petr@2ndquadrant.com> wrote:
> On 22/05/14 06:21, Robert Haas wrote:
>> The main thing I'm not sure about is how to format the message that we
>> write to the shm_mq.  One option is to try to use the good old FEBE
>> protocol.  This doesn't look entirely straightforward, because
>> send_message_to_frontend() assembles the message using pq_sendbyte()
>> and pq_sendstring(), and then sends it out to the client using
>> pq_endmessage() and pq_flush().  This infrastructure assumes that the
>> connection to the frontend is a socket.  It doesn't seem impossible to
>> kludge that infrastructure to be able to send to either a socket or a
>> shm_mq, but I'm not sure whether it's a good idea.  Alternatively, we
>> could devise some other message format specific to this problem; it
>> would probably look a lot like an ErrorData protocol message, but
>> maybe that's doesn't really matter.  Any thoughts?
>
> I played with this a bit already and even have some (very much hacked up)
> prototype based on ErrorData as it seemed better solution than using
> sockets. Obviously some of the ErrorData fields don't really have meaning
> during transport (send_to_client for example) as those must be set based on
> the connection options and I didn't get to making this nicer yet.

I didn't mean the ErrorData structure that the backend uses
internally; it doesn't seem practical to use that because it contains
pointers.  Even if we stored all the data in the DSM (which seems
rather hard without a shared-memory allocator), it might not be mapped
at the same address in both backends.  I meant the libpq wire protocol
used to transmit errors to the client; I guess the message is actually
called ErrorResponse rather than ErrorData:

http://www.postgresql.org/docs/current/static/protocol-message-formats.html

ErrorResponse of course wouldn't contain anything like send_to_client,
as that's a server-internal thing.

>> Another thing to think about is that, most likely, many users of the
>> background worker facility will want to respond to a relayed error by
>> rethrowing it.  That means that whatever format we use to send the
>> error from one process to the other has to be able to be decoded by
>> the receiving process.  That process will probably want to do
>> something like add add a bit more to the context (e.g. "in background
>> worker PID %d") and throw the resulting error preserving the rest of
>> the original fields.  I'm not sure exactly what make sense here, but
>> the point is that ideally the message format should be something that
>> the receiver can rethrow, possibly after tweaking it a bit.
>>
>
> This is one advantage of using ErrorData or ErrorData-like structure based
> messaging, rethrowing is simpler, but I guess if we really needed we could
> provide some conversion api.
>
> One side effect of the rethrowing that maybe deserves a bit of thought is
> that we are going to log the same error from both bgworker and backend if we
> rethrow it.

Yeah, I'm not sure how to handle that.  It probably wouldn't be hard
to make the routine that re-throws the error suppress sending it to
the server log, but that could also be confusing if someone is trying
to follow the progress of a particular session.  Having the background
worker suppress sending it to the  log also seems bad; it has no
guarantee that it will successfully relay the message, or that the
recipient will re-throw it.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company