Re: PQntuples for older version (2.2.1) of psycopg2 with ctypes

On Fri, Feb 19, 2016 at 11:34 PM, Alan Evans <alanwevans@gmail.com> wrote:
> I am using cursor.copy_expert and I would like to be able to use
> cursor.rowcount but most of the systems I am targeting have older psycopg2
> versions on them.
>
> According to the following URL rowcount was updated for use with copy_expert
> in 2.5.3...
> https://psycopg.lighthouseapp.com/projects/62710/tickets/180-return-number-of-rows-for-copy-operations
>
> Inspired by:
> https://github.com/sunlightlabs/psycopg2-ctypes/blob/master/psycopg2ct/_impl/cursor.py
>
> I wonder if it is possible to use ctypes to do this...  Something like this
> maybe:
>
> import psycopg2
> import ctypes
> libpq = ctypes.CDLL('libpq.so.5')
>
> conn = psycopg2.connect(foo)
> cursor = conn.cursor()
> cursor.copy_expert(bar)
>
> rowcount = libpq.PQntuples(???)
>
> I can "read" the psycopg2 source but I am not fluent.  I am having trouble
> working out what 'pgres' is.  Is it the fileno() the backend_process_id()
> etc...
>
> So far all I have manged to do is segfault my python processes.  Woot!
>
> Is 'pgres' exposed by the Cursor or Connection classes somewhere?

Funny hack :)

The pgres is on the cursor:
<https://github.com/psycopg/psycopg2/blob/2_2_1/psycopg/cursor.h#L60>.
Thankfully (at least in 2.2.1 but I guess it never changed) it is
located right after the cur.description pointer, so we can use it to
find the offset pgres offset in the structure:

    import psycopg2
    cnn = psycopg2.connect('')
    cur = cnn.cursor()
    cur.execute("select 1")
    import ctypes, struct, sys
    curdump = ctypes.string_at(id(cur), sys.getsizeof(cur))
    descptr = struct.pack('P', id(cur.description))
    resoff = curdump.find(descptr) + len(descptr)

The offset is fixed for a given build so it won't change at runtime.
You can inspect the cursor after the COPY operation to find the result
structure and pass it to the libpq:

    libpq = ctypes.cdll.LoadLibrary("libpq.so")
    cur.execute("select generate_series(1,42)")  # cur.copy_expert here
    curdump = ctypes.string_at(id(cur), sys.getsizeof(cur))
    resptr = struct.unpack('P', curdump[resoff:resoff+len(descptr)])[0]
    print libpq.PQntuples(resptr)
    # 42

Hope this helps :)

-- Daniele