Обсуждение: connection file descriptors created with identical number after process fork on mac

Hi All,

I'm writing some multi-process code in Python and trying to make sure I
open a new connection for each process. Here's the really cut down code:

"""
import os, time
import psycopg2
from multiprocessing import Pool


def init():
     conn = psycopg2.connect("dbname=...host=localhost")
     print os.getpid(), ' child fd:', conn.fileno()

if __name__=='__main__':
     pool = Pool(initializer=init)
     time.sleep(30)
"""

What's really surpising to me is the output on a mac:

$ python psycopg2_multiprocess.py
44276  child fd: 13
44277  child fd: 13
44278  child fd: 13
44279  child fd: 13

The getpid() output indicates that the connec() call is being made
inside a different process each time, yet the connection appears to
still be using the same fd.

conn.file() is basically (long int)PQsocket(self->pgconn);:
https://github.com/psycopg/psycopg2/blob/master/psycopg/connection_type.c#L898

Is there something I'm missing about file descriptors on Macs or is
something bad happening here?

Chris

Chris Withers <chris@simplistix.co.uk> writes:
> I'm writing some multi-process code in Python and trying to make sure I
> open a new connection for each process. Here's the really cut down code:
> ...
> What's really surpising to me is the output on a mac:

> $ python psycopg2_multiprocess.py
> 44276  child fd: 13
> 44277  child fd: 13
> 44278  child fd: 13
> 44279  child fd: 13

> The getpid() output indicates that the connec() call is being made
> inside a different process each time, yet the connection appears to
> still be using the same fd.

FD numbers are process-local in all flavors of Unix.  The above only
proves that all of these processes had FDs 0..12 open already, which
doesn't seem terribly surprising.

            regards, tom lane

On 04/08/2016 00:20, Tom Lane wrote:
> Chris Withers <chris@simplistix.co.uk> writes:
>> I'm writing some multi-process code in Python and trying to make sure I
>> open a new connection for each process. Here's the really cut down code:
>> ...
>> What's really surpising to me is the output on a mac:
>
>> $ python psycopg2_multiprocess.py
>> 44276  child fd: 13
>> 44277  child fd: 13
>> 44278  child fd: 13
>> 44279  child fd: 13
>
>> The getpid() output indicates that the connec() call is being made
>> inside a different process each time, yet the connection appears to
>> still be using the same fd.
>
> FD numbers are process-local in all flavors of Unix.  The above only
> proves that all of these processes had FDs 0..12 open already, which
> doesn't seem terribly surprising.

Thanks, that's certainly good news!

How can I convince myself, from the client side, that I really have got
a new connection and not somehow ended up with one that been passed on
as part of the fork?

cheers,

Chris

On 8/3/2016 4:25 PM, Chris Withers wrote:
>
> How can I convince myself, from the client side, that I really have
> got a new connection and not somehow ended up with one that been
> passed on as part of the fork?

$ psql -tc "select pg_backend_pid();"
           18635

$ psql -tc "select pg_backend_pid();"
           18665

$ psql -tc "select pg_backend_pid();"
           18727


now, operating system process ID's do get recycled eventually but not
very fast.

--
john r pierce, recycling bits in santa cruz