C# w/ ODBC, 2.1 million list select gives empty DataSet
Hi everyone,
I've been using the ODBC driver for Windows along with my C# (Framework 2.0) for a few years now and it's been working very nicely.
Until now.
One of my customer's database has grown to considerable size over time, and in particular, one report he attemps to do in my software gives an empty report.
Upon investigation, I determined that the SELECT I send to the postgresql gets processed properly and it probably hands over about 2.1 million rows as the response. After that, I use the "Fill" method as instructed in the "HOW TO" section in the ODBC documentation to fill the data into a DataSet object.
After considerable amount of work (almost the same as when I do that same query in PgAdmin III, about 110 seconds), the DataSet gets initialized empty, with zero tables in it. No errors, no exceptions. Everything moves along as if nothing happened.
I just upgraded to the latest ODBC drivers (dated december 2009) and it's not changing my issue.
Am I hitting a limit of some sort? I played with the "ConnectionTimeout" values of OdbcCommand and OdbcDataAdapter, didn't seem to change.
Here's a selected sample of the code I use:
OdbcCommand odcQuery = new OdbcCommand(sSQLQuery);
using (OdbcConnection dbcPostGresConnection = new OdbcConnection(_dpDBSettings.GetConnectionString()))
{
try
{
odcQuery.Connection = dbcPostGresConnection;
dbcPostGresConnection.Open();
switch (emCurrentType)
{
case emSqlQueryType.CREATE:
case emSqlQueryType.DELETE:
case emSqlQueryType.UPDATE:
case emSqlQueryType.INSERT:
case emSqlQueryType.ALTER:
case emSqlQueryType.DROP:
{
_kNumberOfAffectedRecordsFromLastQuery = odcQuery.ExecuteNonQuery();
break;
}
case emSqlQueryType.SELECT:
{
OdbcDataAdapter oddaDatasetFiller = new OdbcDataAdapter(odcQuery);
oddaDatasetFiller.Fill(_dsLastResults);
if (_dsLastResults.Tables.Count > 0)
{
_kNumberOfAffectedRecordsFromLastQuery = _dsLastResults.Tables[0].Rows.Count;
}
break;
}
}
_bLastQuerySucceeded = true;
}
catch... etc all possible catches ahead.
Anyone has a suggestion or ideas? I've searched the web, mailing lists and documentation as I could, nothing seems to be said about this particular issue.
One interseting thing I might add, I was previously using a very old ODBC driver, from 2006. This one caused an exception on the .Fill() method, which was "ERROR [HY000] Out of memory while reading tuples". Now I'm assuming that since I upgraded to the new driver, not having this exception means that I'm not running out of memory anymore, right?
Even more details: with the original 2006 drivers, I could see my application (the .EXE) climb up to 700MB of private bytes/virtual size in Task Manager until eventually crashing and then not releasing that memory. So initially, I was investigating a memory leak issue.
With the new drivers, it still climbs to 700MB or so (makes sense, 2 million rows is quite a bit), and then after the "0 rows" returned, memory is cleared immediately. Makes sense as well, OBDC doesn't carry data anymore, dataset is almost null, garbage collector is having a field day.
Thanks for any suggestions or input from anyone.
Regards,
Laurent Chouinard
I've been using the ODBC driver for Windows along with my C# (Framework 2.0) for a few years now and it's been working very nicely.
Until now.
One of my customer's database has grown to considerable size over time, and in particular, one report he attemps to do in my software gives an empty report.
Upon investigation, I determined that the SELECT I send to the postgresql gets processed properly and it probably hands over about 2.1 million rows as the response. After that, I use the "Fill" method as instructed in the "HOW TO" section in the ODBC documentation to fill the data into a DataSet object.
After considerable amount of work (almost the same as when I do that same query in PgAdmin III, about 110 seconds), the DataSet gets initialized empty, with zero tables in it. No errors, no exceptions. Everything moves along as if nothing happened.
I just upgraded to the latest ODBC drivers (dated december 2009) and it's not changing my issue.
Am I hitting a limit of some sort? I played with the "ConnectionTimeout" values of OdbcCommand and OdbcDataAdapter, didn't seem to change.
Here's a selected sample of the code I use:
OdbcCommand odcQuery = new OdbcCommand(sSQLQuery);
using (OdbcConnection dbcPostGresConnection = new OdbcConnection(_dpDBSettings.GetConnectionString()))
{
try
{
odcQuery.Connection = dbcPostGresConnection;
dbcPostGresConnection.Open();
switch (emCurrentType)
{
case emSqlQueryType.CREATE:
case emSqlQueryType.DELETE:
case emSqlQueryType.UPDATE:
case emSqlQueryType.INSERT:
case emSqlQueryType.ALTER:
case emSqlQueryType.DROP:
{
_kNumberOfAffectedRecordsFromLastQuery = odcQuery.ExecuteNonQuery();
break;
}
case emSqlQueryType.SELECT:
{
OdbcDataAdapter oddaDatasetFiller = new OdbcDataAdapter(odcQuery);
oddaDatasetFiller.Fill(_dsLastResults);
if (_dsLastResults.Tables.Count > 0)
{
_kNumberOfAffectedRecordsFromLastQuery = _dsLastResults.Tables[0].Rows.Count;
}
break;
}
}
_bLastQuerySucceeded = true;
}
catch... etc all possible catches ahead.
Anyone has a suggestion or ideas? I've searched the web, mailing lists and documentation as I could, nothing seems to be said about this particular issue.
One interseting thing I might add, I was previously using a very old ODBC driver, from 2006. This one caused an exception on the .Fill() method, which was "ERROR [HY000] Out of memory while reading tuples". Now I'm assuming that since I upgraded to the new driver, not having this exception means that I'm not running out of memory anymore, right?
Even more details: with the original 2006 drivers, I could see my application (the .EXE) climb up to 700MB of private bytes/virtual size in Task Manager until eventually crashing and then not releasing that memory. So initially, I was investigating a memory leak issue.
With the new drivers, it still climbs to 700MB or so (makes sense, 2 million rows is quite a bit), and then after the "0 rows" returned, memory is cleared immediately. Makes sense as well, OBDC doesn't carry data anymore, dataset is almost null, garbage collector is having a field day.
Thanks for any suggestions or input from anyone.
Regards,
Laurent Chouinard
| Laurent Chouinard laurent.chouinard@sem.ca | SEM inc. 3610 Valiquette, St-Laurent, QC, H4S 1X8 Tel: 514-334-7569 / 1-888-334-7569 Fax: 514-334-5922 |
Вложения
В списке pgsql-odbc по дате отправления:
Предыдущее
От:Дата:
Сообщение: [ psqlodbc-Bugs-1010739 ] Crash in psqlodbc35w.dll v8.4.1 when getting data of type "TEXT" from PSQL Server v8.3.6