Обсуждение: Memory leak in 8.0 JDBC driver?
Before I file a bug, I'd like to ask if the problem is known or if I have done anything wrong. I am running a database stress test against a PostgreSQL 8.0.3 database on a Linux machine. The test program runs on a Windows 2000 client via JDBC, it starts several threads, each of which opens a database connection and issues random statements against the database until the program is interrupted. After about 2 hours, the program always dies with an OutOfMemoryError. I have tried both JRE 1.4.2_02 and 1.5.0_04 from Sun. I would suspect a programming error on my side, except the same program on the same machine runs for days without problems if I run it against other databases (Derby, Firebird, MySql, Oracle, MaxDB). I have got the JDBC3 driver 8.0 Build 312 from http://jdbc.postgresql.org/download.html, and I have also tried downloading and building from source. No matter what I do, memory runs out. - Should I report this as a bug? - I did not want to spam the list with the source of my test program, but I'd happily do so if you think it would help. - Do you have any hints as to how I could debug the problem myself? Can you recommend a memory profiler? Thank you, Laurenz Albe
There are several great commercial products to track down Java memory leaks (I can recommend YourKit). There's also an open-source profiler called JMP (http://www.khelekore.org/jmp/) that also has some memory profiling capabilities. (Actually, it didn't used to, but I mentioned to the maintainer that it would be a great feature to add, and so he did. Very responsive guy). JMP isn't as polished and doesn't make it quite as easy to track down elusive leaks, but still works pretty well. Besides using a memory profiler, you should also check to make sure that ResultSet and Statement objects are always closed. Some drivers may not leak memory if you forget to close those, but some will. Not sure which camp PostgreSQL falls in, but it might explain the different behavior when testing different DB's. -- Mark On Thu, 2005-08-11 at 19:07 +0200, Albe Laurenz wrote: > Before I file a bug, I'd like to ask if the problem is known or if I > have done anything wrong. > > I am running a database stress test against a PostgreSQL 8.0.3 database > on a Linux machine. > The test program runs on a Windows 2000 client via JDBC, it starts > several threads, each > of which opens a database connection and issues random statements > against the database > until the program is interrupted. > > After about 2 hours, the program always dies with an OutOfMemoryError. > > I have tried both JRE 1.4.2_02 and 1.5.0_04 from Sun. > I would suspect a programming error on my side, except the same program > on the same machine > runs for days without problems if I run it against other databases > (Derby, Firebird, MySql, > Oracle, MaxDB). > > I have got the JDBC3 driver 8.0 Build 312 from > http://jdbc.postgresql.org/download.html, > and I have also tried downloading and building from source. > No matter what I do, memory runs out. > > - Should I report this as a bug? > - I did not want to spam the list with the source of my test program, > but I'd happily do > so if you think it would help. > - Do you have any hints as to how I could debug the problem myself? Can > you recommend a > memory profiler? > > Thank you, > Laurenz Albe > > ---------------------------(end of broadcast)--------------------------- > TIP 2: Don't 'kill -9' the postmaster
On Thu, 11 Aug 2005, Albe Laurenz wrote: > I am running a database stress test against a PostgreSQL 8.0.3 database > on a Linux machine. The test program runs on a Windows 2000 client via > JDBC, it starts several threads, each of which opens a database > connection and issues random statements against the database until the > program is interrupted. > > After about 2 hours, the program always dies with an OutOfMemoryError. > > - Should I report this as a bug? > - I did not want to spam the list with the source of my test program, > but I'd happily do so if you think it would help. More information is definitely required. Your test program would be great and the stack trace from your failure wouldn't hurt. Kris Jurka
Are you gracefully closing the resultsets, statements, connections etc... Also .. for 2+hrs., creating new threads ... Cuz that can also kill memory Pl. send me ur code Thanks Nanu -----Original Message----- From: pgsql-jdbc-owner@postgresql.org [mailto:pgsql-jdbc-owner@postgresql.org] On Behalf Of Albe Laurenz Sent: Thursday, August 11, 2005 10:07 AM To: pgsql-jdbc@postgresql.org Subject: [JDBC] Memory leak in 8.0 JDBC driver? Before I file a bug, I'd like to ask if the problem is known or if I have done anything wrong. I am running a database stress test against a PostgreSQL 8.0.3 database on a Linux machine. The test program runs on a Windows 2000 client via JDBC, it starts several threads, each of which opens a database connection and issues random statements against the database until the program is interrupted. After about 2 hours, the program always dies with an OutOfMemoryError. I have tried both JRE 1.4.2_02 and 1.5.0_04 from Sun. I would suspect a programming error on my side, except the same program on the same machine runs for days without problems if I run it against other databases (Derby, Firebird, MySql, Oracle, MaxDB). I have got the JDBC3 driver 8.0 Build 312 from http://jdbc.postgresql.org/download.html, and I have also tried downloading and building from source. No matter what I do, memory runs out. - Should I report this as a bug? - I did not want to spam the list with the source of my test program, but I'd happily do so if you think it would help. - Do you have any hints as to how I could debug the problem myself? Can you recommend a memory profiler? Thank you, Laurenz Albe ---------------------------(end of broadcast)--------------------------- TIP 2: Don't 'kill -9' the postmaster
Please send us the program so that we can see why, or replicate the problem. Dave On 11-Aug-05, at 1:07 PM, Albe Laurenz wrote: > Before I file a bug, I'd like to ask if the problem is known or if I > have done anything wrong. > > I am running a database stress test against a PostgreSQL 8.0.3 > database > on a Linux machine. > The test program runs on a Windows 2000 client via JDBC, it starts > several threads, each > of which opens a database connection and issues random statements > against the database > until the program is interrupted. > > After about 2 hours, the program always dies with an OutOfMemoryError. > > I have tried both JRE 1.4.2_02 and 1.5.0_04 from Sun. > I would suspect a programming error on my side, except the same > program > on the same machine > runs for days without problems if I run it against other databases > (Derby, Firebird, MySql, > Oracle, MaxDB). > > I have got the JDBC3 driver 8.0 Build 312 from > http://jdbc.postgresql.org/download.html, > and I have also tried downloading and building from source. > No matter what I do, memory runs out. > > - Should I report this as a bug? > - I did not want to spam the list with the source of my test program, > but I'd happily do > so if you think it would help. > - Do you have any hints as to how I could debug the problem myself? > Can > you recommend a > memory profiler? > > Thank you, > Laurenz Albe > > ---------------------------(end of > broadcast)--------------------------- > TIP 2: Don't 'kill -9' the postmaster > >
I think that I have found the memory leak, and it is indeed in the JDBC driver. All code references in the following refer to the 8.0-312 JDBC Source. I have an org.postgresql.jdbc3.Jdbc3PreparedStatement containing INSERT INTO PARENT (ID, NAME, NUMMER) VALUES (?, ?, ?) I execute this statement thousands of times with different parameters. After each execution I commit(). Some of these statements fail due to a constraint violation, then I rollback(). Whenever this statement is executed *and fails*, execution reaches org.postgresql.core.v3.QueryExecutorImpl.processResults(), line 1280 where an org.postgresql.util.PSQLWarning is retrieved with the contents: LOG: statement: INSERT INTO PARENT (ID, NAME, NUMMER) VALUES ($1, $2, $3) The handler that handles the warning is the anonymous class in org.postgresql.core.v3.QueryExecutorImpl.sendQueryPreamble(), line 381. This in turn calls org.postgresql.jdbc2.AbstractJdbc2Statement.StatementResultHandler.handl eWarning(), line 191. This causes the SQLWarning to be appended at the end of the warning chain of the prepared statement, as you can see in org.postgresql.jdbc2.AbstractJdbc2Statement.addWarning(), line 546. The problem is that the warning chain is never cleared. According to the documentation of java.sql.Statement.getWarnings(), 'The warning chain is automatically cleared each time a statement is (re)executed' This obviously does not happen. Indeed, when searching the source of the JDBC driver, I cannot find a single reference to the clearWarnings() method, nor is org.postgresql.jdbc2.AbstractJdbc2Statement.warnings ever reset directly. The warnings chain grows endlessly and gobbles up the heap. I guess I should file a bug report for this, right? Yours, Laurenz Albe
On 12-Aug-05, at 11:15 AM, Albe Laurenz wrote: > I think that I have found the memory leak, and it is indeed in the > JDBC > driver. > All code references in the following refer to the 8.0-312 JDBC Source. > > I have an org.postgresql.jdbc3.Jdbc3PreparedStatement containing > INSERT INTO PARENT (ID, NAME, NUMMER) VALUES (?, ?, ?) > I execute this statement thousands of times with different parameters. > After each execution I commit(). > Some of these statements fail due to a constraint violation, then I > rollback(). > > Whenever this statement is executed *and fails*, execution reaches > org.postgresql.core.v3.QueryExecutorImpl.processResults(), line 1280 > where an org.postgresql.util.PSQLWarning is retrieved with the > contents: > LOG: statement: INSERT INTO PARENT (ID, NAME, NUMMER) VALUES ($1, $2, > $3) > > The handler that handles the warning is the anonymous class in > org.postgresql.core.v3.QueryExecutorImpl.sendQueryPreamble(), line > 381. > > This in turn calls > org.postgresql.jdbc2.AbstractJdbc2Statement.StatementResultHandler.han > dl > eWarning(), line 191. > > This causes the SQLWarning to be appended at the end of the warning > chain of > the prepared statement, as you can see in > org.postgresql.jdbc2.AbstractJdbc2Statement.addWarning(), line 546. > > The problem is that the warning chain is never cleared. According > to the > documentation of java.sql.Statement.getWarnings(), > 'The warning chain is automatically cleared each time a statement is > (re)executed' > This obviously does not happen. > > Indeed, when searching the source of the JDBC driver, I cannot find a > single reference > to the clearWarnings() method, nor is > org.postgresql.jdbc2.AbstractJdbc2Statement.warnings > ever reset directly. > > The warnings chain grows endlessly and gobbles up the heap. > > I guess I should file a bug report for this, right? You just did, thanks Dave > > Yours, > Laurenz Albe > > ---------------------------(end of > broadcast)--------------------------- > TIP 2: Don't 'kill -9' the postmaster > >
On Fri, 12 Aug 2005, Albe Laurenz wrote: > I think that I have found the memory leak, and it is indeed in the JDBC > driver. All code references in the following refer to the 8.0-312 JDBC > Source. Yes, you have. I've put fixes into CVS for both the 8.0 and 8.1 branches. > where an org.postgresql.util.PSQLWarning is retrieved with the contents: > LOG: statement: INSERT INTO PARENT (ID, NAME, NUMMER) VALUES ($1, $2, > $3) > The reason we haven't seen reports of this before is that you appear to have enabled statement logging and adjusted client_min_messages to "log" so by default these messages never get to the client. Excellent analysis. Kris Jurka
Kris Jurka wrote: >> I think that I have found the memory leak > > Yes, you have. I've put fixes into CVS for both the 8.0 and > 8.1 branches. Thank you! How frequently are the versions on the download page updated? By the way, as soon as I subscribed and posted to this list, I started receiving spam e-mail :^( Yours, Laurenz Albe
On Tue, 16 Aug 2005, Albe Laurenz wrote: > How frequently are the versions on the download page updated? > I generally put out new releases when an important fix or feature (on the dev branch) has been committed or whenever a reasonable number of minor fixes have built up on the stable branch. Checking the ChangeLog page should give you an approximation of the releaes frequency and checking the commit list archives should give you an idea of the current backlog of fixes. http://jdbc.postgresql.org/changes.html http://gborg.postgresql.org/pipermail/pgjdbc-commit/ Of course anyone is welcome to lobby for a release if they think one is warranted. Kris Jurka