Re: AWS and postgres issues

On Mon, Apr 08, 2013 at 02:09:42PM -0700, David Kerr wrote:
- On Mon, Apr 08, 2013 at 02:14:14PM -0600, Quentin Hartman wrote:
- - What version of pgpool are you using?
- -
- - Are there other commands you have a problem with? I would suspect that the
- - restart is causing the postgres server to go away, pgpool decides to
- - disconnect, and then it has to be manually added back to the cluster.
- - Unless of course you've got automatic failback setup, but even then I would
- - expect that command to do weird things when issued through middleware like
- - pgpool, regardless of what sort of infrastructure you are running on.
- -
- - QH
-
- This is actually from the command line, PgPool isn't involved at all. I just
- mentioned it to give some context.

I had a brief conversation with Quentin offline which indicated that I wasn't
being nearly clear nor direct enough.

I believe that this probelm is specific to AWS+Postgres. (and possibly specific to
VPC+Amazon Linux). Non postgres commands run fine, and even psql works fine.
so far just pg_ctl fails.

I'm running the command directly from an interactive shell, so this is as basic as it gets.

The command runs correctly on the remote server, it just never exits the ssh connection.
specifically I never get: "debug2: channel 0: rcvd close  " as if the packet gets
dropped every time.

I don't believe it's a network hiccup because I can reproduce it every time.

It's likely something with Amazon's infrastructure that's eating it, but whateever
it is, it seems to specifically not like pg_ctl.

Here is what I see happen:
[pgpool@ccpgp05 ~]$ ssh -vvv postgres@10.0.1.30 '/usr/pgsql-9.2/bin/pg_ctl -D /db/pg -m fast restart'
OpenSSH_6.1p1, OpenSSL 1.0.1e-fips 11 Feb 2013
[..snip..]
debug1: Sending command: /usr/pgsql-9.2/bin/pg_ctl -D /db/pg -m fast restart
[..snip..]
waiting for server to shut down.... done
server stopped
server starting
[..snip..]
debug1: client_input_channel_req: channel 0 rtype exit-status reply 0
debug1: client_input_channel_req: channel 0 rtype eow@openssh.com reply 0
debug2: channel 0: rcvd eow
debug2: channel 0: close_read
debug2: channel 0: input open -> closed
^Cdebug1: channel 0: free: client-session, nchannels 1                  # <---- This is where I ^C it
debug3: channel 0: status: The following connections are open:
  #0 client-session (t4 r0 i3/0 o0/0 fd -1/5 cc -1)


Notice the input open -> closed is where it basically hangs

Now look at:
[pgpool@ccpgp05 ~]$ ssh -vvv postgres@10.0.1.30 'ls -ltr'
OpenSSH_6.1p1, OpenSSL 1.0.1e-fips 11 Feb 2013
[..snip..]
debug1: Sending command: ls -ltr
debug2: channel 0: output open -> drain
debug1: channel 0: forcing write
total 8
drwx------ 4 postgres postgres 4096 Apr  4 22:50 9.2
drwx------ 2 postgres postgres 4096 Apr  5 22:58 bin
[..snip..]
debug2: channel 0: input open -> closed
debug2: channel 0: rcvd close
debug3: channel 0: will not send data after close
debug2: channel 0: almost dead
debug2: channel 0: gc: notify user
debug2: channel 0: gc: user detached
debug2: channel 0: send close
debug2: channel 0: is dead
debug2: channel 0: garbage collecting
debug1: channel 0: free: client-session, nchannels 1
debug3: channel 0: status: The following connections are open:
  #0 client-session (t4 r0 i3/0 o3/0 fd -1/-1 cc -1)

Transferred: sent 2456, received 2448 bytes, in 0.0 seconds
Bytes per second: sent 105666.4, received 105322.3
debug1: Exit status 0


Thanks.