Hi,
I got a bug report for txpostgres about NOTIFY events being missed if a
large number of events were sent at the same time. After investigating,
I believe it's a psycopg bug causing the behaviour.
Commit d9c0b8166f9ba17eacb09d5861409ab862447331 removed
conn_notifies_process from pq_fetch for what I guess were good reasons.
The problem is that the async execution patch will call pq_fetch after
an asynchronous query is finished and any notifies received at that
point will not be exposed to the application until the next poll() cycle.
What actually happens is that conn_poll calls _conn_poll_query, which
then calls pq_is_busy. That will check for notifies, but only the ones
that have already been received on the socket at that time. With enough
notifies to span several TCP packets, some of them will remain unread by
the client. Note that the result of _conn_poll_query will be
PSYCO_POLL_OK anyway.
conn_poll then continues to call pq_get_last_result, which calls down to
PQgetResult, causing the remainder of the data to be read from the
socket, along with the pending notifies. However, they won't be parsed
and made available on the Python Connection object.
Attached is a patch that solved the problem for me and a test case that
demonstrates how the connection is missing notifies.
I'm not 100% sure if that's the correct place to put a
conn_notifies_process() call - perhaps it should be in conn_poll inside
the block starting with:
if (res == PSYCO_POLL_OK && self->async && self->async_cursor) {
In any case, it should have a C comment attached, but as I wasn't sure
if the fix is in the right place, I haven't added one yet.
Credit for reporting the bug and preparing the initial test case goes to
Max Walton, which I have taken the liberty of CC'ing.
As an aside: what's the correct process for sending patches to Psycopg?
I see that tickets are tracked in Lighthouse, but the project is still
taking pull requests on GitHub?
Best,
Jan