Re: Connection problem under extreme load.

Tom Lane wrote:

> Interesting.  I *think* (not totally sure) that 'Connection refused'
> here implies that the kernel rejected the connection before the
> postmaster ever had a chance to do anything with it.  The most likely
> reason would probably be that the maximum connection backlog was
> exceeded.  On my system (HPUX) man listen(2) sez
>
>      int listen(int s, int backlog);
>
>      ...
>
>      backlog defines the desirable queue length for pending connections.
>      The actual queue length may be greater than the specified backlog . If
>      a connection request arrives when the queue is full, the client will
>      receive an ETIMEDOUT error.
>
>      backlog is limited to the range of 0 to SOMAXCONN, which is defined in
>      <sys/socket.h>.  SOMAXCONN is currently set to 20.  If any other value
>      is specified, the system automatically assigns the closest value
>      within the range.  A backlog of 0 specifies only 1 pending connection
>      is allowed at any given time.
>
> ETIMEDOUT is not the error you are getting, but that could be a platform
> difference.  In fact the nearest BSD system I have access to says that
> "the client will receive an error with an indication of ECONNREFUSED".
> The same box defines SOMAXCONN as 5, which seems a tad low :-(
>
> So, it would seem your options are
>   (a) recompile your kernel with larger SOMAXCONN, or
>   (b) figure out why the postmaster isn't responding faster.
>
> Offhand, the only performance problem I know of in the postmaster is
> that it does IDENT checks serially --- if you specify ident checks in
> pg_hba.conf, the postmaster will wait for a response from the ident
> server before processing more connection requests.  So if you're using
> IDENT authentication you might want to consider some other answer, or
> else fix that code and send in a patch.
>
> If that's not it, please poke into it further and let us know what you
> find out.
>
>                         regards, tom lane

I think you are correct.  The listen man page on my machine (Sun Solaris)
says:

     If a connection request arrives with  the  queue  full,  the client  will
receive
    an error with an indication of ECONNREFUSED...

The SOMAXCONN field is also 5, which IS a tad low.

Unfortunately, I don't have the ability to rebuild the kernel so this is not
an option.

As to why the postmaster was not responding faster, I think it was because of
the load on the machine.  The load was so heavy, and there were so many
connection requests at the same time, I am not surprised that it could not
keep up.  My test was probably not a realistic load.

I think my best option is to retry the connection when this happens.  I do
wish my kernel would return a different failure, because there really is no
way to distinguish a legitimate ECONNREFUSED (i.e. the server really isn't
listening), versus a backlog queue full situation.

Once again, thank you very much,
Jeff