Обсуждение: HUGE Stack space is gettiing consumed
Hello All, We are getting very strange problem .. Scenario.. ******************************************************** We Have one data field in some table..type of this field Is *TEXT*. We are storing around 1,000 (One Thousand) lines of text In that field. If it's run though separate thread, When we say select <columnname> from <tablename> We are getting *could not receive data from server: No such file or directory * messages without Result If it was in main thread, It Works fine. ******************************************************** Our Analysis:: ******************************************************** It looks like stack space issue.when we allocate stack space Of 1.3 GB to new thread then we will get the Desired Outpt I.e All 1,000 lines without any problem. 1)That String is barely 80 k ..there is no direct Relationship with that size. 2)We can't allocate 1.3 GB,and we also don't have any Recusrsive functions, We would like to know is there any Recursive functions In postgres -PQExec which might be eating all stack space.. Or what cud be the reason form libpq's point of view. ******************************************************** Please give u'r thougts, Thanks in advance for U'r Suggetions.. Prasanna.
On Fri, Apr 14, 2006 at 09:13:26AM +0530, Mavinakuli, Prasanna (STSD) wrote: > Hello All, > > We are getting very strange problem .. > > ******************************************************** > We Have one data field in some table..type of this field > Is *TEXT*. > We are storing around 1,000 (One Thousand) lines of text > In that field. > > If it's run though separate thread, > When we say select <columnname> from <tablename> > We are getting > *could not receive data from server: No such file or directory * > messages without Result > > If it was in main thread, > It Works fine. > ******************************************************** So you have a multithreaded program. Have you setup something so that you don't have two threads trying to do things with libpq at the same time? Because that won't work. Secondly, it can't have anything to do with stack-space because when you run out of stack space you get a segmentation fault, not a nice error. Either post a complete example of the failure (with source code) or perhaps the strace output would be enough. -- Martijn van Oosterhout <kleptog@svana.org> http://svana.org/kleptog/ > Patent. n. Genius is 5% inspiration and 95% perspiration. A patent is a > tool for doing 5% of the work and then sitting around waiting for someone > else to do the other 95% so you can sue them.
Вложения
On Tue, Apr 18, 2006 at 12:31:57PM +0530, Mavinakuli, Prasanna (STSD) wrote: > Hello Martijin, > Thx for u'r Suggetions.. > > Here is Complete source code.. Ok, I can't get it to fail. I cleaned up the code because it wouldn't compile as is (g++ 3.3.5 on Debian). I've put the cleaned up version here: http://svana.org/kleptog/temp/test.cpp You might need to change the QUERY_STRING and the CONNECT_STRING back. Could you try running this to see if you can get it to fail? If it's does fail, could you post the output of: strace -f <program> somewhere (*don't* post it to the list). Somewhere like: http://phpfi.com/ should work well. > Desc about following code:::: > > Code has a function named as queryDB, which makes a connection to our DB > and then tries to query > Particular table which has huge string in it's field. > > If this method called from main thread then it *DOES* work properly. > > But if we make that as routine function for a thread it doesn't work > instead we will get that following status ::: > ( PGRES_FATAL_ERROR ) > ***could not receive data from server: Invalid argument*** Have a nice day, -- Martijn van Oosterhout <kleptog@svana.org> http://svana.org/kleptog/ > Patent. n. Genius is 5% inspiration and 95% perspiration. A patent is a > tool for doing 5% of the work and then sitting around waiting for someone > else to do the other 95% so you can sue them.
Вложения
Hello Martijin,
Thx for u'r Suggetions..
Here is Complete source code..
Desc about following code::::
Code has a function named as queryDB, which makes a connection to our DB
and then tries to query
Particular table which has huge string in it's field.
If this method called from main thread then it *DOES* work properly.
But if we make that as routine function for a thread it doesn't work
instead we will get that following status :::
( PGRES_FATAL_ERROR )
***could not receive data from server: Invalid argument***
******************************CODE*****************************
#include <iostream>
#include "libpq-fe.h"
#include "libpq/libpq-fs.h"
#include <string>
#include <stdlib.h>
#include <netinet/in.h>
#include <arpa/inet.h>
#include <sys/timeb.h>
#include <time.h>
#include <unistd.h>
#include <pwd.h>
#include <sys/timeb.h>
#include <errno.h>
#include <sys/socket.h>
#include <netdb.h>
#include <netinet/in.h>
#include "common/gs/utils/UtilsConsts.h"
#include <utmp.h>
#include <utmpx.h>
#include <sys/stat.h>
#include <cstdlib>
#include <cstdio>
void* queryDB(void* p)
{
cout<<"I Am in Routine Function *** "<<endl;
PGconn *_conn;
PGresult *_res;
char *dbn,*usn;
char *error;
cout<<"i am connecting:"<<endl;
_conn=PQconnectdb("dbname = LOGDB user = sfmdb port=10864 ");
if(PQstatus(_conn)==CONNECTION_OK)
cout<<"connection is ready"<<endl;
dbn=PQdb(_conn);
usn=PQuser(_conn);
if(PQstatus(_conn)==CONNECTION_BAD)
cout<<"the connection is not hapened"<<endl;
string string1,string2,string3;
_res=PQexec(_conn,"select * from throttlingconfig ;");
if (!( PQresultStatus(_res) == PGRES_COMMAND_OK
||PQresultStatus(_res) == PGRES_TUPLES_OK ) )
{
/*If this Function is *NOT* called from main thread then
we will end up by entering here. */
string msg = PQresultErrorMessage(_res);
PQclear(_res);
cout<<"Query Execution is Not OK **********"<<msg<<endl;
}
else
{
/*If this Function is called from main thread then we will
end up by entering here. */
int no_of_tuples,no_of_columns;
char *str1;
string stg;
no_of_columns=PQnfields(_res);
no_of_tuples=PQntuples(_res);
cout<<"no of tuples "<<no_of_tuples<<endl;
cout<<"Number Of Columns "<<no_of_columns<<endl;
for(int u=0;u<no_of_tuples;u++)
{
for(int y=0;y<no_of_columns;y++)
{
str1=PQgetvalue(_res,u,y);
cout<<"*************Printing *** "<<u<<"th Row
"<<y<<"th Column****************"<<endl;
cout<<str1<<endl;
}
cout<<endl;
}
PQclear(_res);
}
PQfinish(_conn);
return NULL;
}
int main()
{
//size_t _stackSize(512 * 1024 * 2);
//calling queryDB from other thread ..DOESN'T Work here
size_t _stackSize(512 * 1024 * 2 * 20) ;
pthread_attr_t attr;
pthread_attr_init ( &attr ) ;
pthread_t thread_t;
int ret = pthread_attr_setstacksize( &attr, _stackSize) ;
pthread_attr_setdetachstate(&attr, PTHREAD_CREATE_JOINABLE);
int retVal = pthread_create(&thread_t, &attr,queryDB,NULL );
int rc,status;
rc = pthread_join(thread_t, (void **)&status);
//calling queryDB from main thread WORKS here..
queryDB(NULL);
return 1;
}
Thx a lot,
And looking for u'r reply,
Prasanna.
-----Original Message-----
From: Martijn van Oosterhout [mailto:kleptog@svana.org]
Sent: Saturday, April 15, 2006 7:22 PM
To: Mavinakuli, Prasanna (STSD)
Cc: pgsql-general@postgresql.org
Subject: Re: [GENERAL] HUGE Stack space is gettiing consumed
On Fri, Apr 14, 2006 at 09:13:26AM +0530, Mavinakuli, Prasanna (STSD)
wrote:
> Hello All,
>
> We are getting very strange problem ..
>
> ********************************************************
> We Have one data field in some table..type of this field Is *TEXT*.
> We are storing around 1,000 (One Thousand) lines of text In that
> field.
>
> If it's run though separate thread,
> When we say select <columnname> from <tablename> We are getting *could
> not receive data from server: No such file or directory * messages
> without Result
>
> If it was in main thread,
> It Works fine.
> ********************************************************
So you have a multithreaded program. Have you setup something so that
you don't have two threads trying to do things with libpq at the same
time? Because that won't work.
Secondly, it can't have anything to do with stack-space because when you
run out of stack space you get a segmentation fault, not a nice error.
Either post a complete example of the failure (with source code) or
perhaps the strace output would be enough.
--
Martijn van Oosterhout <kleptog@svana.org> http://svana.org/kleptog/
> Patent. n. Genius is 5% inspiration and 95% perspiration. A patent is
> a tool for doing 5% of the work and then sitting around waiting for
> someone else to do the other 95% so you can sue them.
On Thu, Apr 20, 2006 at 08:20:30PM +0530, Mavinakuli, Prasanna (STSD) wrote: > > Hello Martijin, > > I am not able to make it ..Code what u sent is *not* giving desired > result.. > That is -when we try to fetch huge data in a thread routine it says > "could not receive data from server: Error 0" > If it was in main thread then it goes thorugh..I.e It's able to get the > huge data. Well, I don't understand. There's no interesting difference between the code I sent and the code you sent... > Will send the tusc output (stack trace) Not stack trace, strace which is a system call trace. Don't forget -f to trace the threads also. -- Martijn van Oosterhout <kleptog@svana.org> http://svana.org/kleptog/ > From each according to his ability. To each according to his ability to litigate.
Вложения
Hello Martijin, I am not able to make it ..Code what u sent is *not* giving desired result.. That is -when we try to fetch huge data in a thread routine it says "could not receive data from server: Error 0" If it was in main thread then it goes thorugh..I.e It's able to get the huge data. Environment.. Postgres Version:7.4.3 Compiler:aCC 5.0 OS:HP-UX PARISC 11.23 Code has put in the following link.. http://phpfi.com/113850 Will send the tusc output (stack trace) Thx, Prasanna. -----Original Message----- From: Martijn van Oosterhout [mailto:kleptog@svana.org] Sent: Tuesday, April 18, 2006 2:46 PM To: Mavinakuli, Prasanna (STSD) Cc: pgsql-general@postgresql.org Subject: Re: [GENERAL] HUGE Stack space is gettiing consumed On Tue, Apr 18, 2006 at 12:31:57PM +0530, Mavinakuli, Prasanna (STSD) wrote: > Hello Martijin, > Thx for u'r Suggetions.. > > Here is Complete source code.. Ok, I can't get it to fail. I cleaned up the code because it wouldn't compile as is (g++ 3.3.5 on Debian). I've put the cleaned up version here: http://svana.org/kleptog/temp/test.cpp You might need to change the QUERY_STRING and the CONNECT_STRING back. Could you try running this to see if you can get it to fail? If it's does fail, could you post the output of: strace -f <program> somewhere (*don't* post it to the list). Somewhere like: http://phpfi.com/ should work well. > Desc about following code:::: > > Code has a function named as queryDB, which makes a connection to our > DB and then tries to query Particular table which has huge string in > it's field. > > If this method called from main thread then it *DOES* work properly. > > But if we make that as routine function for a thread it doesn't work > instead we will get that following status ::: > ( PGRES_FATAL_ERROR ) > ***could not receive data from server: Invalid argument*** Have a nice day, -- Martijn van Oosterhout <kleptog@svana.org> http://svana.org/kleptog/ > Patent. n. Genius is 5% inspiration and 95% perspiration. A patent is > a tool for doing 5% of the work and then sitting around waiting for > someone else to do the other 95% so you can sue them.
On Sat, Apr 22, 2006 at 08:45:49AM +0530, Mavinakuli, Prasanna (STSD) wrote: > Hello Martijin, > > I have one more doudt ..while building libpq.so we are not specifying > > --enable-thread-safety > > Will it make any difference..? Well, if you don't, the libpq won't know anything about threads and may indeed have problems. I can't see anything in your other output that would suggest that the system is doing anything wrong, so if you havn't got --enable-thread-safety on, perhaps the main and the subthreads are using a different C library or something obscure. I'd try that, see if it helps, -- Martijn van Oosterhout <kleptog@svana.org> http://svana.org/kleptog/ > From each according to his ability. To each according to his ability to litigate.
Вложения
Hello Martijin, I have one more doudt ..while building libpq.so we are not specifying --enable-thread-safety Will it make any difference..? Thx, Prasanna. -----Original Message----- From: Mavinakuli, Prasanna (STSD) Sent: Friday, April 21, 2006 5:46 PM To: 'Martijn van Oosterhout' Subject: RE: [GENERAL] HUGE Stack space is gettiing consumed Hello Martijin, Here is System call trace .. Details http://phpfi.com/114019 Working V/s Non-Working http://phpfi.com/114021 (deleted obv art from above one) Please go through it.. Thx, Prasanna. -----Original Message----- From: Martijn van Oosterhout [mailto:kleptog@svana.org] Sent: Thursday, April 20, 2006 9:00 PM To: Mavinakuli, Prasanna (STSD) Cc: pgsql-general@postgresql.org Subject: Re: [GENERAL] HUGE Stack space is gettiing consumed On Thu, Apr 20, 2006 at 08:20:30PM +0530, Mavinakuli, Prasanna (STSD) wrote: > > Hello Martijin, > > I am not able to make it ..Code what u sent is *not* giving desired > result.. > That is -when we try to fetch huge data in a thread routine it says > "could not receive data from server: Error 0" > If it was in main thread then it goes thorugh..I.e It's able to get > the huge data. Well, I don't understand. There's no interesting difference between the code I sent and the code you sent... > Will send the tusc output (stack trace) Not stack trace, strace which is a system call trace. Don't forget -f to trace the threads also. -- Martijn van Oosterhout <kleptog@svana.org> http://svana.org/kleptog/ > From each according to his ability. To each according to his ability to litigate.