Обсуждение: crash bug on closed connection
I have been looking into this crash for a while now, and I finally have a good repro.
After digging through it with wireshark I observed this
client tries to send a query
retransmit query
retransmit query
...
client sends TCP [RST],[ACK]
Now it has crashed, so we restart it again and begin another connection successfully.
It seems that the connection has dropped, but the client was never informed, and it doesn't handle this gracefully.
I reproed it like this:
On my mac running postgres server:
Setup port forwarding to emulate a proxy or firewall problem
ssh -L [public ip]:5433:localhost:5432 -N localhost
On my windows machine:
Connect to port 5433 on my mac
Run a query
On my mac:
Kill the ssh proxy
sighub postgres
open the ssh proxy again
On my windows machine:
Run another query (was never informed that the connection dropped)
Crash in
I have been looking into this crash for a while now, and I finally have a good repro.
After digging through it with wireshark I observed this
client tries to send a query
retransmit query
retransmit query
...
client sends TCP [RST],[ACK]
Now it has crashed, so we restart it again and begin another connection successfully.
It seems that the connection has dropped, but the client was never informed, and it doesn't handle this gracefully.
I reproed it like this:
On my mac running postgres server:
Setup port forwarding to emulate a proxy or firewall problem
ssh -L [public ip]:5433:localhost:5432 -N localhost
On my windows machine:
Connect to port 5433 on my mac
Run a query
On my mac:
Kill the ssh proxy
sighup postgres
open the ssh proxy again
On my windows machine:
Run another query (machine still thinks connection is ok)
Crash in psqlodbcw.dll
> psqlodbc35w.dll!Exec_with_parameters_resolved(StatementClass_ * stmt=0x0a6f4fa0, int * exec_end=0x083bf3dc) Line 606 + 0x1b bytes C
psqlodbc35w.dll!PGAPI_Execute(void * hstmt=0x0a6f4fa0, unsigned short flag=1) Line 1147 + 0xd bytes C
psqlodbc35w.dll!SQLExecute(void * StatementHandle=0x0a6f4fa0) Line 386 + 0xe bytes C
odbc32.dll!_SQLExecute@4() + 0x2c1 bytes
> if (sscanf(cmd , "UPDATE %d", &count) == 1) // here cmd is junk but not null (0xddddddd)
Some of these steps might not be necessary. It might could be accomplished with a kill -9 on postgres. I tried running Postgres on Windows, and force killing it, but then the client always knew the connection was down.
When this occurs "in the wild", it is usually related to a timeout, so the fix that was checked in recently for setting tcpKeepAlive might be a workaround.
On Mon, Oct 13, 2014 at 4:44 PM, Jade Koskela <jkoskela0@gmail.com> wrote:
I have been looking into this crash for a while now, and I finally have a good repro.After digging through it with wireshark I observed thisclient tries to send a queryretransmit queryretransmit query...client sends TCP [RST],[ACK]Now it has crashed, so we restart it again and begin another connection successfully.It seems that the connection has dropped, but the client was never informed, and it doesn't handle this gracefully.I reproed it like this:On my mac running postgres server:Setup port forwarding to emulate a proxy or firewall problemssh -L [public ip]:5433:localhost:5432 -N localhostOn my windows machine:Connect to port 5433 on my macRun a queryOn my mac:Kill the ssh proxysighup postgresopen the ssh proxy againOn my windows machine:Run another query (machine still thinks connection is ok)Crash in psqlodbcw.dll> psqlodbc35w.dll!Exec_with_parameters_resolved(StatementClass_ * stmt=0x0a6f4fa0, int * exec_end=0x083bf3dc) Line 606 + 0x1b bytes Cpsqlodbc35w.dll!PGAPI_Execute(void * hstmt=0x0a6f4fa0, unsigned short flag=1) Line 1147 + 0xd bytes Cpsqlodbc35w.dll!SQLExecute(void * StatementHandle=0x0a6f4fa0) Line 386 + 0xe bytes Codbc32.dll!_SQLExecute@4() + 0x2c1 bytes> if (sscanf(cmd , "UPDATE %d", &count) == 1) // here cmd is junk but not null (0xddddddd)
Also the mac/pc isn't relevant information in the repro that I posted. It might as well just be client/server.
On Mon, Oct 13, 2014 at 10:26 PM, Jade Koskela <jkoskela0@gmail.com> wrote:
Some of these steps might not be necessary. It might could be accomplished with a kill -9 on postgres. I tried running Postgres on Windows, and force killing it, but then the client always knew the connection was down.When this occurs "in the wild", it is usually related to a timeout, so the fix that was checked in recently for setting tcpKeepAlive might be a workaround.On Mon, Oct 13, 2014 at 4:44 PM, Jade Koskela <jkoskela0@gmail.com> wrote:I have been looking into this crash for a while now, and I finally have a good repro.After digging through it with wireshark I observed thisclient tries to send a queryretransmit queryretransmit query...client sends TCP [RST],[ACK]Now it has crashed, so we restart it again and begin another connection successfully.It seems that the connection has dropped, but the client was never informed, and it doesn't handle this gracefully.I reproed it like this:On my mac running postgres server:Setup port forwarding to emulate a proxy or firewall problemssh -L [public ip]:5433:localhost:5432 -N localhostOn my windows machine:Connect to port 5433 on my macRun a queryOn my mac:Kill the ssh proxysighup postgresopen the ssh proxy againOn my windows machine:Run another query (machine still thinks connection is ok)Crash in psqlodbcw.dll> psqlodbc35w.dll!Exec_with_parameters_resolved(StatementClass_ * stmt=0x0a6f4fa0, int * exec_end=0x083bf3dc) Line 606 + 0x1b bytes Cpsqlodbc35w.dll!PGAPI_Execute(void * hstmt=0x0a6f4fa0, unsigned short flag=1) Line 1147 + 0xd bytes Cpsqlodbc35w.dll!SQLExecute(void * StatementHandle=0x0a6f4fa0) Line 386 + 0xe bytes Codbc32.dll!_SQLExecute@4() + 0x2c1 bytes> if (sscanf(cmd , "UPDATE %d", &count) == 1) // here cmd is junk but not null (0xddddddd)
Here is the end of the log.
I've done some debugging, when CC_send_query_append is called here, it correctly identifies that there was an error, and exits the function.
res = CC_send_query_append(conn, self->stmt_with_params, qryi, qflag, SC_get_ancestor(self), appendq); (statement.c:2069)
The result which is returned is not handled correctly (statement.c:2078)
if (appendq)
{
QResultClass *qres, *nres;
if ( res->rstatus == PORES_EMPTY_QUERY && !res->next)
res = NULL;
for (qres = res; qres;)
{
if (qres->command && strnicmp(qres->command, fetch_cmd, 5) == 0)
{
res = qres;
break;
}
nres = qres->next;
if (nres && QR_get_notice(qres) != NULL)
{
if (QR_command_successful(nres) &&
QR_command_nonfatal(qres))
{
QR_set_rstatus(nres, PORES_NONFATAL_ERROR);
}
QR_add_notice(nres, QR_get_notice(qres));
}
qres->next = NULL;
QR_Destructor(qres);
qres = nres;
}
}
There is only a single result, with result.next == NULL.
This part of the code calls the destructor on the only result, but res is still left pointing at it. After that fun things happen.
It gets a long way before it actually crashes in Exec_with_parameters_resolved.
On Mon, Oct 13, 2014 at 10:27 PM, Jade Koskela <jkoskela0@gmail.com> wrote:
Also the mac/pc isn't relevant information in the repro that I posted. It might as well just be client/server.On Mon, Oct 13, 2014 at 10:26 PM, Jade Koskela <jkoskela0@gmail.com> wrote:Some of these steps might not be necessary. It might could be accomplished with a kill -9 on postgres. I tried running Postgres on Windows, and force killing it, but then the client always knew the connection was down.When this occurs "in the wild", it is usually related to a timeout, so the fix that was checked in recently for setting tcpKeepAlive might be a workaround.On Mon, Oct 13, 2014 at 4:44 PM, Jade Koskela <jkoskela0@gmail.com> wrote:I have been looking into this crash for a while now, and I finally have a good repro.After digging through it with wireshark I observed thisclient tries to send a queryretransmit queryretransmit query...client sends TCP [RST],[ACK]Now it has crashed, so we restart it again and begin another connection successfully.It seems that the connection has dropped, but the client was never informed, and it doesn't handle this gracefully.I reproed it like this:On my mac running postgres server:Setup port forwarding to emulate a proxy or firewall problemssh -L [public ip]:5433:localhost:5432 -N localhostOn my windows machine:Connect to port 5433 on my macRun a queryOn my mac:Kill the ssh proxysighup postgresopen the ssh proxy againOn my windows machine:Run another query (machine still thinks connection is ok)Crash in psqlodbcw.dll> psqlodbc35w.dll!Exec_with_parameters_resolved(StatementClass_ * stmt=0x0a6f4fa0, int * exec_end=0x083bf3dc) Line 606 + 0x1b bytes Cpsqlodbc35w.dll!PGAPI_Execute(void * hstmt=0x0a6f4fa0, unsigned short flag=1) Line 1147 + 0xd bytes Cpsqlodbc35w.dll!SQLExecute(void * StatementHandle=0x0a6f4fa0) Line 386 + 0xe bytes Codbc32.dll!_SQLExecute@4() + 0x2c1 bytes> if (sscanf(cmd , "UPDATE %d", &count) == 1) // here cmd is junk but not null (0xddddddd)
Вложения
Hi Jade,
Could you please try the attached patch?
regards,
Hiroshi Inoue
On 2014/10/15 14:38, Jade Koskela wrote:
> Here is the end of the log.
> I've done some debugging, when CC_send_query_append is called here, it
> correctly identifies that there was an error, and exits the function.
>
> res = CC_send_query_append(conn, self->stmt_with_params, qryi, qflag,
> SC_get_ancestor(self), appendq); (statement.c:2069)
>
> The result which is returned is not handled correctly (statement.c:2078)
>
> if (appendq)
> {
> QResultClass *qres, *nres;
>
> if ( res->rstatus == PORES_EMPTY_QUERY && !res->next)
> res = NULL;
>
> for (qres = res; qres;)
> {
> if (qres->command && strnicmp(qres->command, fetch_cmd, 5) == 0)
> {
> res = qres;
> break;
> }
> nres = qres->next;
> if (nres && QR_get_notice(qres) != NULL)
> {
> if (QR_command_successful(nres) &&
> QR_command_nonfatal(qres))
> {
> QR_set_rstatus(nres, PORES_NONFATAL_ERROR);
> }
> QR_add_notice(nres, QR_get_notice(qres));
> }
> qres->next = NULL;
> QR_Destructor(qres);
> qres = nres;
> }
> }
>
> There is only a single result, with result.next == NULL.
> This part of the code calls the destructor on the only result, but res
> is still left pointing at it. After that fun things happen.
> It gets a long way before it actually crashes in
> Exec_with_parameters_resolved.
Вложения
(2014/10/16 4:23), Inoue, Hiroshi wrote: > Hi Jade, > > Could you please try the attached patch? Oops the patch seems to contain *color* attributes. Pleaae try the attached patch for this mail instead. regards, Hiroshi Inoue -- I am using the free version of SPAMfighter. SPAMfighter has removed 12760 of my spam emails to date. Get the free SPAMfighter here: http://www.spamfighter.com/len Do you have a slow PC? Try a Free scan http://www.spamfighter.com/SLOW-PCfighter?cid=sigen
Вложения
Yeah that fixes it. Thanks! I tried to fix it myself, but it was just a hack.
Did this make the cutoff for the next release?
Jade Koskela
On Wed, Oct 15, 2014 at 4:59 PM, Inoue, Hiroshi <inoue@tpf.co.jp> wrote:
(2014/10/16 4:23), Inoue, Hiroshi wrote:Hi Jade,
Could you please try the attached patch?
Oops the patch seems to contain *color* attributes.
Pleaae try the attached patch for this mail instead.
regards,
Hiroshi Inoue
--
I am using the free version of SPAMfighter.
SPAMfighter has removed 12760 of my spam emails to date.
Get the free SPAMfighter here: http://www.spamfighter.com/len
Do you have a slow PC? Try a Free scan http://www.spamfighter.com/SLOW-PCfighter?cid=sigen
Sorry to revive this, but I don't think this patch ever made it into the repository.
On Thu, Oct 16, 2014 at 10:28 AM, Jade Koskela <jkoskela0@gmail.com> wrote:
Yeah that fixes it. Thanks! I tried to fix it myself, but it was just a hack.Did this make the cutoff for the next release?Jade KoskelaOn Wed, Oct 15, 2014 at 4:59 PM, Inoue, Hiroshi <inoue@tpf.co.jp> wrote:(2014/10/16 4:23), Inoue, Hiroshi wrote:Hi Jade,
Could you please try the attached patch?
Oops the patch seems to contain *color* attributes.
Pleaae try the attached patch for this mail instead.
regards,
Hiroshi Inoue
--
I am using the free version of SPAMfighter.
SPAMfighter has removed 12760 of my spam emails to date.
Get the free SPAMfighter here: http://www.spamfighter.com/len
Do you have a slow PC? Try a Free scan http://www.spamfighter.com/SLOW-PCfighter?cid=sigen