Обсуждение: Caching driver on pgFoundry?
JDBC team, I may have missed some stuff over the holiday weekend, but why is the new caching wrapper on pgfoundry instead of on jdbc.postgresql.org? -- Josh Berkus PostgreSQL @ Sun San Francisco
Josh Berkus wrote: > I may have missed some stuff over the holiday weekend, but why is the new > caching wrapper on pgfoundry instead of on jdbc.postgresql.org? See the discussion here: http://archives.postgresql.org/pgsql-jdbc/2007-07/msg00137.php continuing in August here: http://archives.postgresql.org/pgsql-jdbc/2007-08/msg00006.php -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com
Heikki, > See the discussion here: > http://archives.postgresql.org/pgsql-jdbc/2007-07/msg00137.php > continuing in August here: > http://archives.postgresql.org/pgsql-jdbc/2007-08/msg00006.php I understand why it's a wrapper. I don't understand why the wrapper isn't at jdbc.postgresql.org. Putting it on pgfoundry, completely separate from all the JDBC drivers, is pretty much a guarentee that nobody will ever download it. Or is it moving to jdbc.postgresql.org once it's tested? Just so everyone is clear on why this is important & urgent ... we published a benchmark[1] using the caching driver, which is the only published benchmark PostgreSQL has. This benchmark has generated a huge amount of interest in PostgreSQL as an alternative to Oracle[2], and is very important to driving the adoption of PostgreSQL *especially* amoung J2EE developers. So it would be nice to see the caching wrapper represented as "official" unless there's something technically wrong with it. [1]http://www.spec.org/jAppServer2004/results/res2007q3/jAppServer2004-20070703-00073.html [2]http://www.informationweek.com/showArticle.jhtml?articleID=201001901 -- Josh Berkus PostgreSQL @ Sun San Francisco
Josh Berkus wrote: > I understand why it's a wrapper. I don't understand why the wrapper isn't at > jdbc.postgresql.org. Putting it on pgfoundry, completely separate from all > the JDBC drivers, is pretty much a guarentee that nobody will ever download > it. I don't have a problem with linking from jdbc.postgresql.org to the wrapper. And other pooling and caching implementations as well while you're at it. > Just so everyone is clear on why this is important & urgent ... we published a > benchmark[1] using the caching driver, which is the only published benchmark > PostgreSQL has. This benchmark has generated a huge amount of interest in > PostgreSQL as an alternative to Oracle[2], and is very important to driving > the adoption of PostgreSQL *especially* amoung J2EE developers. So it would > be nice to see the caching wrapper represented as "official" unless there's > something technically wrong with it. Can't you use DBCP or some other open source statement cache implementation that's in a more mature state? Or actually, why don't you have a statement cache in Sun Application Server like the competitors? ;-) -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com
Heikki, > I don't have a problem with linking from jdbc.postgresql.org to the > wrapper. And other pooling and caching implementations as well while > you're at it. Only one I can think of is Sequoia. > Can't you use DBCP or some other open source statement cache > implementation that's in a more mature state? Unfortunately, no. The benchmark is already out. > Or actually, why don't you > have a statement cache in Sun Application Server like the competitors? ;-) Don't even get me started. -- Josh Berkus PostgreSQL @ Sun San Francisco
On Wed, 5 Sep 2007, Josh Berkus wrote: >> Can't you use DBCP or some other open source statement cache >> implementation that's in a more mature state? > > Unfortunately, no. The benchmark is already out. > But that benchmark was run with a different caching implementation than the wrapper version, so I'm not sure how that's relevant. When Sun chose to use unpublished and unreviewed code for the benchmark they got themselves in a little bind and I'm not sure it's our job to bail them out by publishing and advertising code that we're not confident in. Heikki, Oliver, and myself did not believe the code used by Sun in the benchmark was correct in the general case so it was rejected for inclusion in the core driver. Dave/Lazlo have since started a new implemention on pgfoundry, but that was never discussed with the JDBC list or submitted for inclusion. To satisfy the benchmark requirements, Sun should publish the code/driver actually used in the benchmark somewhere on Sun's website and, if honest, should recommend that people don't use it. From there we should try to gather some consensus on whether PG needs its own statement cache implementation and then rerun the benchmarks with it or some other implementation. Kris Jurka
On 5-Sep-07, at 12:21 PM, Heikki Linnakangas wrote: > Josh Berkus wrote: >> I understand why it's a wrapper. I don't understand why the >> wrapper isn't at >> jdbc.postgresql.org. Putting it on pgfoundry, completely separate >> from all >> the JDBC drivers, is pretty much a guarentee that nobody will ever >> download >> it. > The drivers will end up there eventually anyway. And we can link it to the jdbc home page. Dave > I don't have a problem with linking from jdbc.postgresql.org to the > wrapper. And other pooling and caching implementations as well while > you're at it. > >> Just so everyone is clear on why this is important & urgent ... we >> published a >> benchmark[1] using the caching driver, which is the only published >> benchmark >> PostgreSQL has. This benchmark has generated a huge amount of >> interest in >> PostgreSQL as an alternative to Oracle[2], and is very important >> to driving >> the adoption of PostgreSQL *especially* amoung J2EE developers. >> So it would >> be nice to see the caching wrapper represented as "official" >> unless there's >> something technically wrong with it. > > Can't you use DBCP or some other open source statement cache > implementation that's in a more mature state? Or actually, why > don't you > have a statement cache in Sun Application Server like the > competitors? ;-) > > -- > Heikki Linnakangas > EnterpriseDB http://www.enterprisedb.com > > ---------------------------(end of > broadcast)--------------------------- > TIP 4: Have you searched our list archives? > > http://archives.postgresql.org
On 5-Sep-07, at 1:27 PM, Kris Jurka wrote: > > > On Wed, 5 Sep 2007, Josh Berkus wrote: > >>> Can't you use DBCP or some other open source statement cache >>> implementation that's in a more mature state? >> >> Unfortunately, no. The benchmark is already out. >> > > But that benchmark was run with a different caching implementation > than the wrapper version, so I'm not sure how that's relevant. > When Sun chose to use unpublished and unreviewed code for the > benchmark they got themselves in a little bind and I'm not sure > it's our job to bail them out by publishing and advertising code > that we're not confident in. Heikki, Oliver, and myself did not > believe the code used by Sun in the benchmark was correct in the > general case so it was rejected for inclusion in the core driver. So as I understand it the objection to the caching implementation is that statement caching belongs in the app server ? If this is the case then I would argue that having caching in the appserver is a great idea for everyone using an appserver it does not help the rest of the world that doesn't use an app server. Is there a technical argument here ? Dave > Dave/Lazlo have since started a new implemention on pgfoundry, but > that was never discussed with the JDBC list or submitted for > inclusion. > > To satisfy the benchmark requirements, Sun should publish the code/ > driver actually used in the benchmark somewhere on Sun's website > and, if honest, should recommend that people don't use it. From > there we should try to gather some consensus on whether PG needs > its own statement cache implementation and then rerun the > benchmarks with it or some other implementation. > > Kris Jurka > > ---------------------------(end of > broadcast)--------------------------- > TIP 4: Have you searched our list archives? > > http://archives.postgresql.org
On Wed, 5 Sep 2007, Dave Cramer wrote: > > On 5-Sep-07, at 1:27 PM, Kris Jurka wrote: > >> But that benchmark was run with a different caching implementation than the >> wrapper version, so I'm not sure how that's relevant. When Sun chose to >> use unpublished and unreviewed code for the benchmark they got themselves >> in a little bind and I'm not sure it's our job to bail them out by >> publishing and advertising code that we're not confident in. Heikki, >> Oliver, and myself did not believe the code used by Sun in the benchmark >> was correct in the general case so it was rejected for inclusion in the >> core driver. > > So as I understand it the objection to the caching implementation is that > statement caching belongs in the app server ? No, the objection above is that the code you originally submitted was not good enough to be included in the driver regardless of whether it should be in the driver or app server. If we aren't using the code that you originally submitted then the argument that we need to publish something for benchmark compliance is silly because the requirement is you publish the code/driver used, not a somewhat similar implementation. > If this is the case then I would argue that having caching in the appserver > is a great idea for everyone using an appserver it does not help the rest of > the world that doesn't use an app server. This is the consensus I'd like to see built before we spend a lot of time on something. Heikki doesn't believe we should do this at all as it is covered by DBCP and app server pools. You clearly believe we need this and I'm sort of on the fence (I recall that's where Oliver is as well, but I don't claim to speak for him). > Is there a technical argument here ? > No, this is just a recap of the previous events from my perspective. Kris Jurka
On 5-Sep-07, at 2:12 PM, Kris Jurka wrote: > > > On Wed, 5 Sep 2007, Dave Cramer wrote: > >> >> On 5-Sep-07, at 1:27 PM, Kris Jurka wrote: >> >>> But that benchmark was run with a different caching >>> implementation than the wrapper version, so I'm not sure how >>> that's relevant. When Sun chose to use unpublished and >>> unreviewed code for the benchmark they got themselves in a little >>> bind and I'm not sure it's our job to bail them out by publishing >>> and advertising code that we're not confident in. Heikki, >>> Oliver, and myself did not believe the code used by Sun in the >>> benchmark was correct in the general case so it was rejected for >>> inclusion in the core driver. >> >> So as I understand it the objection to the caching implementation >> is that statement caching belongs in the app server ? > > No, the objection above is that the code you originally submitted > was not good enough to be included in the driver regardless of > whether it should be in the driver or app server. If we aren't > using the code that you originally submitted then the argument that > we need to publish something for benchmark compliance is silly > because the requirement is you publish the code/driver used, not a > somewhat similar implementation. As I recall the events. The only objection to code that I submitted for inclusion was Heikki's objection as to where caching belonged. The proof of concept code was clearly not production and was not meant for inclusion it was meant to generate this argument. As nobody argued vehemently at the time I went ahead with a production version. > >> If this is the case then I would argue that having caching in the >> appserver is a great idea for everyone using an appserver it does >> not help the rest of the world that doesn't use an app server. > > This is the consensus I'd like to see built before we spend a lot > of time on something. Heikki doesn't believe we should do this at > all as it is covered by DBCP and app server pools. Well DBCP has it's issues. It is very large and not the fastest piece of code in the world. > You clearly believe we need this and I'm sort of on the fence (I > recall that's where Oliver is as well, but I don't claim to speak > for him). So we basically have one nay vote holding this up ? Dave > >> Is there a technical argument here ? >> > > No, this is just a recap of the previous events from my perspective. > > Kris Jurka
On Wed, 5 Sep 2007, Dave Cramer wrote: > On 5-Sep-07, at 2:12 PM, Kris Jurka wrote: > >> You clearly believe we need this and I'm sort of on the fence (I recall >> that's where Oliver is as well, but I don't claim to speak for him). > > So we basically have one nay vote holding this up ? > But we only have one yes vote pushing it forward. At least of the four of us, I haven't gone back to the archives to see if anyone else weighed in on the discussion. Kris Jurka
On 5-Sep-07, at 2:25 PM, Kris Jurka wrote: > > > On Wed, 5 Sep 2007, Dave Cramer wrote: > >> On 5-Sep-07, at 2:12 PM, Kris Jurka wrote: >> >>> You clearly believe we need this and I'm sort of on the fence (I >>> recall that's where Oliver is as well, but I don't claim to speak >>> for him). >> >> So we basically have one nay vote holding this up ? >> > > But we only have one yes vote pushing it forward. At least of the > four of us, I haven't gone back to the archives to see if anyone > else weighed in on the discussion. > I think you can consider Josh Berkus, Laszlo, Paul Vandenbogaard to have voted in favour of it. > Kris Jurka
posted my ideas on august 9. seen no reaction. I think that's a yes vote. Not sure if it counts though. To restate: apps servers have statement caching build in because at the time it was created there were (almost) no drivers out there implementing it. Guess Suns driver is one of the few that does not have an implementation. From an architectural point of view I feel a statement is part of a connection. Therefore it belongs in a connections. However I do admit there are many ways to implement this view. I know that adding an extra jar file will reduce its acceptance in the ISV community. Been working with different ISVs for almost 10 years now and every element they include in their suite will in general be included in their Q&A tests. One more element of the suite they support means many more tests. It makes it more difficult (this to some extent might be perception, but it is something that does matter to them) to do it so they will be more reluctant to support it. The wrapper is an extra layer that needs to be crossed while "creating" a statement from the pool. However I doubt that it will make a significant (negative) impact when compared to the all-in-one approach. The real net effect of using statement caching is the reduction of work on the database side. Finally it is not just appsservers out there. Lots if J2SE application exist in at least the Telco segment. Would love to see these guys to adopt a free database instead of Oracle that is all over that place. --Paul On 5-sep-2007, at 20:25, Kris Jurka wrote: > > > On Wed, 5 Sep 2007, Dave Cramer wrote: > >> On 5-Sep-07, at 2:12 PM, Kris Jurka wrote: >> >>> You clearly believe we need this and I'm sort of on the fence (I >>> recall that's where Oliver is as well, but I don't claim to speak >>> for him). >> >> So we basically have one nay vote holding this up ? >> > > But we only have one yes vote pushing it forward. At least of the > four of us, I haven't gone back to the archives to see if anyone > else weighed in on the discussion. > > Kris Jurka > > ---------------------------(end of > broadcast)--------------------------- > TIP 9: In versions below 8.0, the planner will ignore your desire to > choose an index scan if your joining column's datatypes do not > match ------------------------------------------------------------------------ --------------------- Paul van den Bogaard Paul.vandenBogaard@sun.com CIE -- Collaboration and ISV Engineering, Opensource Engineering group Sun Microsystems, Inc phone: +31 334 515 918 Saturnus 1 extentsion: x (70)15918 3824 ME Amersfoort mobile: +31 651 913 354 The Netherlands fax: +31 334 515 001
Kris, > But that benchmark was run with a different caching implementation than > the wrapper version, so I'm not sure how that's relevant. When Sun > chose to use unpublished and unreviewed code for the benchmark they got > themselves in a little bind and I'm not sure it's our job to bail them > out by publishing and advertising code that we're not confident in. This isn't for *Sun*, it's for PostgreSQL. Do you have any idea how many times a month I field the question "Is there a benchmark for PostgreSQL?" -- --Josh Josh Berkus PostgreSQL @ Sun San Francisco
Kris, > But we only have one yes vote pushing it forward. At least of the four > of us, I haven't gone back to the archives to see if anyone else weighed > in on the discussion. Assuming I have a vote, I vote to push it forward *unless* there is a technical objection to code quality, for the following reasons: 1) The benchmark publication strongly benefits the whole PostgreSQL project and will hopefully lead to expanding our JDBC contributor pool; 2) Since most other DBMSes (including DB2, MySQL and Oracle) have caching in their drivers this makes PostgreSQL more competitive for Java projects; 3) Not all Java users are using appservers, so it's a useful feature for those still using JSP and other non-appserver implementations. -- --Josh Josh Berkus PostgreSQL @ Sun San Francisco
Kris Jurka wrote: > On Wed, 5 Sep 2007, Dave Cramer wrote: >> If this is the case then I would argue that having caching in the >> appserver is a great idea for everyone using an appserver it does not >> help the rest of the world that doesn't use an app server. > > This is the consensus I'd like to see built before we spend a lot of > time on something. Heikki doesn't believe we should do this at all as > it is covered by DBCP and app server pools. To be clear, I think it's nice to have one more statement cache implementation around, with friendly BSD license. I encourage you to keep working on it, regardless of how this discussion ends. But let's not bundle it with the PostgreSQL JDBC driver, it's clearly a separate component. For applications that already have another connection pool / statement cache implementation, it would be just extra baggage. And as a separate project, you can use it with other DBMSs as well. Or bundle it with an app server :). Another point of view is to think about the release cycles of the JDBC driver and the caching wrapper. If a bug is found in the wrapper, and it's bundled with the JDBC driver, we would have to release a new version of the bundle, forcing an upgrade of both components even if you only use one. If we add a new feature to the wrapper, is that going to be backpatched to say the 8.1 branch? What if you're running 8.1 server and driver, and you want to take advantage of a new wrapper feature? The wrapper has enough merit to live a happy and successful life as a project of its own. I don't care if it's a separate pgfoundry project, or just a separate CVS directory within jdbc project and separate builds; the main point is that it should be a separate jar file with a separate release cycle. If we add a link to the latest version of the wrapper jar from jdbc.postgresql.org download page, people will find it regardless of where the development happens. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com
Dave Cramer wrote: > As I recall the events. The only objection to code that I submitted for > inclusion was Heikki's objection as to where caching belonged. I objected to not having a wrapper at all and using the driver's statements directly. That produces semantically incorrect behaviour, that's unacceptable for the main driver. If you're fixing that great but it's quite a different beast to the original implementation and I haven't seen a new version of it yet. The "is it part of the driver or not" question is more of a maintenance issue than anything. If it is packaged separately it can be maintained more easily by whoever cares about it (you, Josh, whoever). If it's part of the driver you're going to run into the barrier that whenever it needs an update you will need to build consensus amongst the JDBC maintainers, which so far you don't have a very good record of doing. I still have not seen an argument as to why it needs to be part of the driver at all, and "not part of the driver" certainly seems simpler and more flexible to me. Did you finish those benchmarks of in-driver versus out-of-driver implementations yet? -O
Kris Jurka wrote: > This is the consensus I'd like to see built before we spend a lot of > time on something. Heikki doesn't believe we should do this at all as > it is covered by DBCP and app server pools. You clearly believe we need > this and I'm sort of on the fence (I recall that's where Oliver is as > well, but I don't claim to speak for him). I think it belongs outside the driver, but I could be convinced otherwise if Dave etc. presents a good technical argument as to why it is easier to implement within the driver. Haven't seen that yet. -O
Oliver Jowett wrote: > The "is it part of the driver or not" question is more of a > maintenance issue than anything. If it is packaged separately it can > be maintained more easily by whoever cares about it (you, Josh, > whoever). If it's part of the driver you're going to run into the > barrier that whenever it needs an update you will need to build > consensus amongst the JDBC maintainers, which so far you don't have a > very good record of doing. How about making it a separate package, but offering a combined download or jar with both packages on the drivers download page. And a README including instructions on how to use / activate caching. Like: postgresql-8.2.506.jdbc4.jar and caching-123.jar combined as: postgresql-8.2.506.jdbc4.caching-123.jar The only consensus needed would be who has to build and upload the combined jar when one of the packages gets updated. That would combine the benefits of having it obviously available for anyone who downloads the driver (unlike a completly separate project on an otherwise unrelated site), and keeping it separated for easy maintenance. Till
> How about making it a separate package, but offering a combined download > or jar with both packages on the drivers download page. And a README > including instructions on how to use / activate caching. Hey, if we can get the same performance, I'm all for this. Thanks for suggestion! -- --Josh Josh Berkus PostgreSQL @ Sun San Francisco
> If this is the case then I would argue that having caching in the > appserver is a great idea for everyone using an appserver it does not > help the rest of the world that doesn't use an app server. We have a product that doesn't use app server, still we need a cached connection pool. In this case this is an Oracle backend, so we started using the cached pool implementation found in the Oracle jdbc driver -- assuming that Oracle will do caching/pooling best for its Connections. The fact is that we've been having some issues with the Oracle cached pool implementation, so I started to extensively test the Apache DBCP and the issues seems to have gone away. Apart from parabolic example above, I think that from a "global" perspective, it would be much more reasonable to help the Apache people solve any issue which someone using the Commons DBCP with PostgreSQL might have than doing your own little house work excercise of implementing yet-another-ccp. Thanks Peter On 9/5/07, Dave Cramer <pg@fastcrypt.com> wrote: > > On 5-Sep-07, at 1:27 PM, Kris Jurka wrote: > > > > > > > On Wed, 5 Sep 2007, Josh Berkus wrote: > > > >>> Can't you use DBCP or some other open source statement cache > >>> implementation that's in a more mature state? > >> > >> Unfortunately, no. The benchmark is already out. > >> > > > > But that benchmark was run with a different caching implementation > > than the wrapper version, so I'm not sure how that's relevant. > > When Sun chose to use unpublished and unreviewed code for the > > benchmark they got themselves in a little bind and I'm not sure > > it's our job to bail them out by publishing and advertising code > > that we're not confident in. Heikki, Oliver, and myself did not > > believe the code used by Sun in the benchmark was correct in the > > general case so it was rejected for inclusion in the core driver. > > So as I understand it the objection to the caching implementation is > that statement caching belongs in the app server ? > > If this is the case then I would argue that having caching in the > appserver is a great idea for everyone using an appserver it does not > help the rest of the world that doesn't use an app server. > > Is there a technical argument here ? > > Dave > > > Dave/Lazlo have since started a new implemention on pgfoundry, but > > that was never discussed with the JDBC list or submitted for > > inclusion. > > > > To satisfy the benchmark requirements, Sun should publish the code/ > > driver actually used in the benchmark somewhere on Sun's website > > and, if honest, should recommend that people don't use it. From > > there we should try to gather some consensus on whether PG needs > > its own statement cache implementation and then rerun the > > benchmarks with it or some other implementation. > > > > Kris Jurka > > > > ---------------------------(end of > > broadcast)--------------------------- > > TIP 4: Have you searched our list archives? > > > > http://archives.postgresql.org > > > ---------------------------(end of broadcast)--------------------------- > TIP 2: Don't 'kill -9' the postmaster >
> Been working with different ISVs for almost 10 > years now and every element they include in their suite will in > general be included in their Q&A tests. One more element of the suite > they support means many more tests. It makes it more difficult (this > to some extent might be perception, but it is something that does > matter to them) to do it so they will be more reluctant to support it. ISV will not extra-test the driver(s) which comes with the database, will they? (I mean, they don't test driver's separately by default.) If you, for example, bundle the Commons DBCP with the JDBC driver, this argument becomes pretty much empty. Thanks Peter On 9/5/07, Paul van den Bogaard <Paul.Vandenbogaard@sun.com> wrote: > posted my ideas on august 9. seen no reaction. I think that's a yes > vote. Not sure if it counts though. > To restate: > > apps servers have statement caching build in because at the time it > was created there were (almost) no drivers out there implementing it. > Guess Suns driver is one of the few that does not have an > implementation. > > From an architectural point of view I feel a statement is part of a > connection. Therefore it belongs in a connections. However I do admit > there are many ways to implement this view. > > I know that adding an extra jar file will reduce its acceptance in > the ISV community. Been working with different ISVs for almost 10 > years now and every element they include in their suite will in > general be included in their Q&A tests. One more element of the suite > they support means many more tests. It makes it more difficult (this > to some extent might be perception, but it is something that does > matter to them) to do it so they will be more reluctant to support it. > > The wrapper is an extra layer that needs to be crossed while > "creating" a statement from the pool. However I doubt that it will > make a significant (negative) impact when compared to the all-in-one > approach. The real net effect of using statement caching is the > reduction of work on the database side. > > Finally it is not just appsservers out there. Lots if J2SE > application exist in at least the Telco segment. Would love to see > these guys to adopt a free database instead of Oracle that is all > over that place. > > > --Paul > > > On 5-sep-2007, at 20:25, Kris Jurka wrote: > > > > > > > On Wed, 5 Sep 2007, Dave Cramer wrote: > > > >> On 5-Sep-07, at 2:12 PM, Kris Jurka wrote: > >> > >>> You clearly believe we need this and I'm sort of on the fence (I > >>> recall that's where Oliver is as well, but I don't claim to speak > >>> for him). > >> > >> So we basically have one nay vote holding this up ? > >> > > > > But we only have one yes vote pushing it forward. At least of the > > four of us, I haven't gone back to the archives to see if anyone > > else weighed in on the discussion. > > > > Kris Jurka > > > > ---------------------------(end of > > broadcast)--------------------------- > > TIP 9: In versions below 8.0, the planner will ignore your desire to > > choose an index scan if your joining column's datatypes do not > > match > > ------------------------------------------------------------------------ > --------------------- > Paul van den Bogaard > Paul.vandenBogaard@sun.com > CIE -- Collaboration and ISV Engineering, Opensource Engineering group > > Sun Microsystems, Inc phone: +31 > 334 515 918 > Saturnus 1 > extentsion: x (70)15918 > 3824 ME Amersfoort mobile: +31 > 651 913 354 > The Netherlands > fax: +31 334 515 001 > > > ---------------------------(end of broadcast)--------------------------- > TIP 9: In versions below 8.0, the planner will ignore your desire to > choose an index scan if your joining column's datatypes do not > match >
The mail below didn't make onto the list. Reason: I sent with the wrong FROM address. Why do I have several e-mail addresses? Because the e-mail address "peter.kovacs.1.0rc@gmail.com" was not accepted/handled/seen/whatever by the postgres mail list server. (Sorry for my having a troubled life.) ---------- Forwarded message ---------- From: Peter Kovacs <peter.kovacs.1.0rc@gmail.com> Date: Sep 6, 2007 9:38 AM Subject: Re: [JDBC] Caching driver on pgFoundry? To: Dave Cramer <pg@fastcrypt.com> Cc: Kris Jurka <books@ejurka.com>, Josh Berkus <josh@agliodbs.com>, Heikki Linnakangas <heikki@enterprisedb.com>, pgsql-jdbc@postgresql.org > If this is the case then I would argue that having caching in the > appserver is a great idea for everyone using an appserver it does not > help the rest of the world that doesn't use an app server. We have a product that doesn't use app server, still we need a cached connection pool. In this case this is an Oracle backend, so we started using the cached pool implementation found in the Oracle jdbc driver -- assuming that Oracle will do caching/pooling best for its Connections. The fact is that we've been having some issues with the Oracle cached pool implementation, so I started to extensively test the Apache DBCP and the issues seems to have gone away. Apart from parabolic example above, I think that from a "global" perspective, it would be much more reasonable to help the Apache people solve any issue which someone using the Commons DBCP with PostgreSQL might have than doing your own little house work excercise of implementing yet-another-ccp. Thanks Peter On 9/5/07, Dave Cramer <pg@fastcrypt.com> wrote: > > On 5-Sep-07, at 1:27 PM, Kris Jurka wrote: > > > > > > > On Wed, 5 Sep 2007, Josh Berkus wrote: > > > >>> Can't you use DBCP or some other open source statement cache > >>> implementation that's in a more mature state? > >> > >> Unfortunately, no. The benchmark is already out. > >> > > > > But that benchmark was run with a different caching implementation > > than the wrapper version, so I'm not sure how that's relevant. > > When Sun chose to use unpublished and unreviewed code for the > > benchmark they got themselves in a little bind and I'm not sure > > it's our job to bail them out by publishing and advertising code > > that we're not confident in. Heikki, Oliver, and myself did not > > believe the code used by Sun in the benchmark was correct in the > > general case so it was rejected for inclusion in the core driver. > > So as I understand it the objection to the caching implementation is > that statement caching belongs in the app server ? > > If this is the case then I would argue that having caching in the > appserver is a great idea for everyone using an appserver it does not > help the rest of the world that doesn't use an app server. > > Is there a technical argument here ? > > Dave > > > Dave/Lazlo have since started a new implemention on pgfoundry, but > > that was never discussed with the JDBC list or submitted for > > inclusion. > > > > To satisfy the benchmark requirements, Sun should publish the code/ > > driver actually used in the benchmark somewhere on Sun's website > > and, if honest, should recommend that people don't use it. From > > there we should try to gather some consensus on whether PG needs > > its own statement cache implementation and then rerun the > > benchmarks with it or some other implementation. > > > > Kris Jurka > > > > ---------------------------(end of > > broadcast)--------------------------- > > TIP 4: Have you searched our list archives? > > > > http://archives.postgresql.org > > > ---------------------------(end of broadcast)--------------------------- > TIP 2: Don't 'kill -9' the postmaster >
Due to painful remnants of my troubled childhood, the message below haven't made it to the list. ---------- Forwarded message ---------- From: Peter Kovacs <peter.kovacs.1.0rc@gmail.com> Date: Sep 6, 2007 9:51 AM Subject: Re: [JDBC] Caching driver on pgFoundry? To: Paul van den Bogaard <Paul.Vandenbogaard@sun.com> Cc: Kris Jurka <books@ejurka.com>, Dave Cramer <pg@fastcrypt.com>, Josh Berkus <josh@agliodbs.com>, Heikki Linnakangas <heikki@enterprisedb.com>, pgsql-jdbc@postgresql.org > Been working with different ISVs for almost 10 > years now and every element they include in their suite will in > general be included in their Q&A tests. One more element of the suite > they support means many more tests. It makes it more difficult (this > to some extent might be perception, but it is something that does > matter to them) to do it so they will be more reluctant to support it. ISV will not extra-test the driver(s) which comes with the database, will they? (I mean, they don't test driver's separately by default.) If you, for example, bundle the Commons DBCP with the JDBC driver, this argument becomes pretty much empty. Thanks Peter On 9/5/07, Paul van den Bogaard <Paul.Vandenbogaard@sun.com> wrote: > posted my ideas on august 9. seen no reaction. I think that's a yes > vote. Not sure if it counts though. > To restate: > > apps servers have statement caching build in because at the time it > was created there were (almost) no drivers out there implementing it. > Guess Suns driver is one of the few that does not have an > implementation. > > From an architectural point of view I feel a statement is part of a > connection. Therefore it belongs in a connections. However I do admit > there are many ways to implement this view. > > I know that adding an extra jar file will reduce its acceptance in > the ISV community. Been working with different ISVs for almost 10 > years now and every element they include in their suite will in > general be included in their Q&A tests. One more element of the suite > they support means many more tests. It makes it more difficult (this > to some extent might be perception, but it is something that does > matter to them) to do it so they will be more reluctant to support it. > > The wrapper is an extra layer that needs to be crossed while > "creating" a statement from the pool. However I doubt that it will > make a significant (negative) impact when compared to the all-in-one > approach. The real net effect of using statement caching is the > reduction of work on the database side. > > Finally it is not just appsservers out there. Lots if J2SE > application exist in at least the Telco segment. Would love to see > these guys to adopt a free database instead of Oracle that is all > over that place. > > > --Paul > > > On 5-sep-2007, at 20:25, Kris Jurka wrote: > > > > > > > On Wed, 5 Sep 2007, Dave Cramer wrote: > > > >> On 5-Sep-07, at 2:12 PM, Kris Jurka wrote: > >> > >>> You clearly believe we need this and I'm sort of on the fence (I > >>> recall that's where Oliver is as well, but I don't claim to speak > >>> for him). > >> > >> So we basically have one nay vote holding this up ? > >> > > > > But we only have one yes vote pushing it forward. At least of the > > four of us, I haven't gone back to the archives to see if anyone > > else weighed in on the discussion. > > > > Kris Jurka > > > > ---------------------------(end of > > broadcast)--------------------------- > > TIP 9: In versions below 8.0, the planner will ignore your desire to > > choose an index scan if your joining column's datatypes do not > > match > > ------------------------------------------------------------------------ > --------------------- > Paul van den Bogaard > Paul.vandenBogaard@sun.com > CIE -- Collaboration and ISV Engineering, Opensource Engineering group > > Sun Microsystems, Inc phone: +31 > 334 515 918 > Saturnus 1 > extentsion: x (70)15918 > 3824 ME Amersfoort mobile: +31 > 651 913 354 > The Netherlands > fax: +31 334 515 001 > > > ---------------------------(end of broadcast)--------------------------- > TIP 9: In versions below 8.0, the planner will ignore your desire to > choose an index scan if your joining column's datatypes do not > match >
Paul van den Bogaard wrote: > Finally it is not just appsservers out there. Lots if J2SE application > exist in at least the Telco segment. Would love to see these guys to > adopt a free database instead of Oracle that is all over that place. Going off at a bit of a tangent here.. OpenCloud does telco infrastructure software and we package postgresql with our product to do persistence of long-lived state (deployed services, cluster configuration, customer provisioning data etc). The statement caching stuff is not really an issue either way for what we do, since we already intelligently reuse statements in our persistance layer. One thing we've found is that our customers don't really care about database setup for this sort of use, they want it to "just work". In fact we package Derby instead of PostgreSQL with our development versions just so there's one less thing for the developer to set up. So even if we cared about statement caching it wouldn't really matter if it was together with the driver or a separate package as we'd still end up doing all the packaging together with the application and testing of the combined product. While what we do is essentially an appserver (JAIN SLEE) I suspect the same applies to standalone java telco apps, you're going to be doing DB integration as part of your app not as something that gets plugged in afterwards, so I'm not sure how relevant having it in the standard driver is. -O
if the caching & pooling stuff is in the same jar file as the rest of the JDBC driver than my point is empty. If it is (can be seen as) an extra element that it is very likely it will mean extra work for the Q&A department of the ISV. Test suite is likely to become on platform A, test with and test without on platform B, test with and test without continues till all platforms are covered. In general one more element translates in more permutations. --Paul On 6-sep-2007, at 9:51, Peter Kovacs wrote: >> Been working with different ISVs for almost 10 >> years now and every element they include in their suite will in >> general be included in their Q&A tests. One more element of the suite >> they support means many more tests. It makes it more difficult (this >> to some extent might be perception, but it is something that does >> matter to them) to do it so they will be more reluctant to support >> it. > > ISV will not extra-test the driver(s) which comes with the database, > will they? (I mean, they don't test driver's separately by default.) > If you, for example, bundle the Commons DBCP with the JDBC driver, > this argument becomes pretty much empty. > > Thanks > Peter > > On 9/5/07, Paul van den Bogaard <Paul.Vandenbogaard@sun.com> wrote: >> posted my ideas on august 9. seen no reaction. I think that's a yes >> vote. Not sure if it counts though. >> To restate: >> >> apps servers have statement caching build in because at the time it >> was created there were (almost) no drivers out there implementing it. >> Guess Suns driver is one of the few that does not have an >> implementation. >> >> From an architectural point of view I feel a statement is part of a >> connection. Therefore it belongs in a connections. However I do admit >> there are many ways to implement this view. >> >> I know that adding an extra jar file will reduce its acceptance in >> the ISV community. Been working with different ISVs for almost 10 >> years now and every element they include in their suite will in >> general be included in their Q&A tests. One more element of the suite >> they support means many more tests. It makes it more difficult (this >> to some extent might be perception, but it is something that does >> matter to them) to do it so they will be more reluctant to support >> it. >> >> The wrapper is an extra layer that needs to be crossed while >> "creating" a statement from the pool. However I doubt that it will >> make a significant (negative) impact when compared to the all-in-one >> approach. The real net effect of using statement caching is the >> reduction of work on the database side. >> >> Finally it is not just appsservers out there. Lots if J2SE >> application exist in at least the Telco segment. Would love to see >> these guys to adopt a free database instead of Oracle that is all >> over that place. >> >> >> --Paul >> >> >> On 5-sep-2007, at 20:25, Kris Jurka wrote: >> >>> >>> >>> On Wed, 5 Sep 2007, Dave Cramer wrote: >>> >>>> On 5-Sep-07, at 2:12 PM, Kris Jurka wrote: >>>> >>>>> You clearly believe we need this and I'm sort of on the fence (I >>>>> recall that's where Oliver is as well, but I don't claim to speak >>>>> for him). >>>> >>>> So we basically have one nay vote holding this up ? >>>> >>> >>> But we only have one yes vote pushing it forward. At least of the >>> four of us, I haven't gone back to the archives to see if anyone >>> else weighed in on the discussion. >>> >>> Kris Jurka >>> >>> ---------------------------(end of >>> broadcast)--------------------------- >>> TIP 9: In versions below 8.0, the planner will ignore your desire to >>> choose an index scan if your joining column's datatypes do not >>> match >> >> --------------------------------------------------------------------- >> --- >> --------------------- >> Paul van den Bogaard >> Paul.vandenBogaard@sun.com >> CIE -- Collaboration and ISV Engineering, Opensource Engineering >> group >> >> Sun Microsystems, Inc phone: +31 >> 334 515 918 >> Saturnus 1 >> extentsion: x (70)15918 >> 3824 ME Amersfoort mobile: +31 >> 651 913 354 >> The Netherlands >> fax: +31 334 515 001 >> >> >> ---------------------------(end of >> broadcast)--------------------------- >> TIP 9: In versions below 8.0, the planner will ignore your desire to >> choose an index scan if your joining column's datatypes do not >> match >> ------------------------------------------------------------------------ --------------------- Paul van den Bogaard Paul.vandenBogaard@sun.com CIE -- Collaboration and ISV Engineering, Opensource Engineering group Sun Microsystems, Inc phone: +31 334 515 918 Saturnus 1 extentsion: x (70)15918 3824 ME Amersfoort mobile: +31 651 913 354 The Netherlands fax: +31 334 515 001
Paul van den Bogaard wrote: > if the caching & pooling stuff is in the same jar file as the rest of > the JDBC driver than my point is empty. If it is (can be seen as) an > extra element that it is very likely it will mean extra work for the Q&A > department of the ISV. > Test suite is likely to become > > on platform A, test with and test without > on platform B, test with and test without > > continues till all platforms are covered. > > In general one more element translates in more permutations. I don't get that. If you need the caching, you're going distribute the caching component with your software, and you're going to test it *with* the caching. Why would you test it without it? On the other hand, if you want to support your software with and without the caching, you should test it in both modes, regardless of the packaging. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com
Seen this too. However I did spend a significant amount of time tuning these applications since they have to deliver a certain throughput. Statement caching is definitely helpful in those cases. For the ones you describe could it be that the dbase layer is just storage and not part of the (critical) performance path? If the database related work is critical in performance/throughput than caching is a welcome technology. I found this matters for the lower level apps to establish call setups, check balance on pre paid, and these basic thing were every tick translates into a tiny bit of money. Unfortunately there are an awful lot of these tiny little bits. This is what makes it so interesting from a performance tuning perspective that is. And yes some of the telco ISVs use Times Ten to do it all in memory. In that case everything is cached. Mmmm could be a nice extension for PG... --Paul On 6-sep-2007, at 11:14, Oliver Jowett wrote: > Paul van den Bogaard wrote: > >> Finally it is not just appsservers out there. Lots if J2SE >> application exist in at least the Telco segment. Would love to see >> these guys to adopt a free database instead of Oracle that is all >> over that place. > > Going off at a bit of a tangent here.. OpenCloud does telco > infrastructure software and we package postgresql with our product > to do persistence of long-lived state (deployed services, cluster > configuration, customer provisioning data etc). The statement > caching stuff is not really an issue either way for what we do, > since we already intelligently reuse statements in our persistance > layer. > > One thing we've found is that our customers don't really care about > database setup for this sort of use, they want it to "just work". > In fact we package Derby instead of PostgreSQL with our development > versions just so there's one less thing for the developer to set > up. So even if we cared about statement caching it wouldn't really > matter if it was together with the driver or a separate package as > we'd still end up doing all the packaging together with the > application and testing of the combined product. > > While what we do is essentially an appserver (JAIN SLEE) I suspect > the same applies to standalone java telco apps, you're going to be > doing DB integration as part of your app not as something that gets > plugged in afterwards, so I'm not sure how relevant having it in > the standard driver is. > > -O ------------------------------------------------------------------------ --------------------- Paul van den Bogaard Paul.vandenBogaard@sun.com CIE -- Collaboration and ISV Engineering, Opensource Engineering group Sun Microsystems, Inc phone: +31 334 515 918 Saturnus 1 extentsion: x (70)15918 3824 ME Amersfoort mobile: +31 651 913 354 The Netherlands fax: +31 334 515 001
Paul van den Bogaard wrote: > For the ones you > describe could it be that the dbase layer is just storage and not part > of the (critical) performance path? Right, in our case we keep all state in memory as the primary representation (and maybe replicated across a cluster) and some specific state is also asynchronously written to disk via a RDBMS so that it survives a cluster restart. Performance of the write-back step is still important as it affects things like our MTTR from a complete database+cluster failure, but it does not directly affect the call throughput / latency. > If the database related work is critical in performance/throughput than > caching is a welcome technology. Well no not necessarily because *statement caching does nothing that you can't do directly in your application anyway*. For example in our case we would get no additional benefit from statement caching because we (as a JDBC client) already hold on to statement objects that we know we're going to reuse, i.e. we're already doing our own caching, and smarter than a generic layer can do because we have better knowledge of how we use our queries. Statement caching is really a "go faster" switch when you don't want to (or cannot) optimize the application code, or where the exact query strings are unpredictable. But again it's not something that *has* to be implemented in the driver, as far as I can tell there's nothing there that can't be done by a layer between the client and the real driver. -O
On 6-sep-2007, at 11:40, Oliver Jowett wrote: > Paul van den Bogaard wrote: > >> For the ones you describe could it be that the dbase layer is just >> storage and not part of the (critical) performance path? > > Right, in our case we keep all state in memory as the primary > representation (and maybe replicated across a cluster) and some > specific state is also asynchronously written to disk via a RDBMS > so that it survives a cluster restart. Performance of the write- > back step is still important as it affects things like our MTTR > from a complete database+cluster failure, but it does not directly > affect the call throughput / latency. > >> If the database related work is critical in performance/throughput >> than caching is a welcome technology. > > Well no not necessarily because *statement caching does nothing > that you can't do directly in your application anyway*. For example > in our case we would get no additional benefit from statement > caching because we (as a JDBC client) already hold on to statement > objects that we know we're going to reuse, i.e. we're already doing > our own caching, and smarter than a generic layer can do because we > have better knowledge of how we use our queries. If all ISVs would do it intelligently than I would have been out of a job :-). I definitely agree. There is only one who can (should) know best and that is the application writer. However I found this is a rare situation. On the other hand if you do not need this most optimal situation than indeed it would be very helpfull if this layer in between would do it for you. > > Statement caching is really a "go faster" switch when you don't > want to (or cannot) optimize the application code, or where the > exact query strings are unpredictable. But again it's not something > that *has* to be implemented in the driver, as far as I can tell > there's nothing there that can't be done by a layer between the > client and the real driver. I think from an implementation point of view you can build in and/or directly on top the JDBC layer. From a performance point of view I do not expect a significant advantage for one or the other either. --Paul > > -O ------------------------------------------------------------------------ --------------------- Paul van den Bogaard Paul.vandenBogaard@sun.com CIE -- Collaboration and ISV Engineering, Opensource Engineering group Sun Microsystems, Inc phone: +31 334 515 918 Saturnus 1 extentsion: x (70)15918 3824 ME Amersfoort mobile: +31 651 913 354 The Netherlands fax: +31 334 515 001
You have a point here. --Paul On 6-sep-2007, at 11:26, Heikki Linnakangas wrote: > Paul van den Bogaard wrote: >> if the caching & pooling stuff is in the same jar file as the rest of >> the JDBC driver than my point is empty. If it is (can be seen as) an >> extra element that it is very likely it will mean extra work for >> the Q&A >> department of the ISV. >> Test suite is likely to become >> >> on platform A, test with and test without >> on platform B, test with and test without >> >> continues till all platforms are covered. >> >> In general one more element translates in more permutations. > > I don't get that. If you need the caching, you're going distribute the > caching component with your software, and you're going to test it > *with* > the caching. Why would you test it without it? > > On the other hand, if you want to support your software with and > without > the caching, you should test it in both modes, regardless of the > packaging. > > -- > Heikki Linnakangas > EnterpriseDB http://www.enterprisedb.com ------------------------------------------------------------------------ --------------------- Paul van den Bogaard Paul.vandenBogaard@sun.com CIE -- Collaboration and ISV Engineering, Opensource Engineering group Sun Microsystems, Inc phone: +31 334 515 918 Saturnus 1 extentsion: x (70)15918 3824 ME Amersfoort mobile: +31 651 913 354 The Netherlands fax: +31 334 515 001
Simon Riggs wrote: > Sun chose to use Blowfish, but could have chosen others for the > benchmark. We shouldn't go down a route with the software that is driven > because of an external, unrelated decision. Blowfish? Is that a nickname for Sun Java System Application Server? -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com
On Wed, 2007-09-05 at 21:20 +0100, Heikki Linnakangas wrote: > Kris Jurka wrote: > > On Wed, 5 Sep 2007, Dave Cramer wrote: > >> If this is the case then I would argue that having caching in the > >> appserver is a great idea for everyone using an appserver it does not > >> help the rest of the world that doesn't use an app server. > > > > This is the consensus I'd like to see built before we spend a lot of > > time on something. Heikki doesn't believe we should do this at all as > > it is covered by DBCP and app server pools. > > To be clear, I think it's nice to have one more statement cache > implementation around, with friendly BSD license. I encourage you to > keep working on it, regardless of how this discussion ends. > > But let's not bundle it with the PostgreSQL JDBC driver, it's clearly a > separate component. For applications that already have another > connection pool / statement cache implementation, it would be just extra > baggage. And as a separate project, you can use it with other DBMSs as > well. Or bundle it with an app server :). > > Another point of view is to think about the release cycles of the JDBC > driver and the caching wrapper. If a bug is found in the wrapper, and > it's bundled with the JDBC driver, we would have to release a new > version of the bundle, forcing an upgrade of both components even if you > only use one. If we add a new feature to the wrapper, is that going to > be backpatched to say the 8.1 branch? What if you're running 8.1 server > and driver, and you want to take advantage of a new wrapper feature? > > The wrapper has enough merit to live a happy and successful life as a > project of its own. I don't care if it's a separate pgfoundry project, > or just a separate CVS directory within jdbc project and separate > builds; the main point is that it should be a separate jar file with a > separate release cycle. If we add a link to the latest version of the > wrapper jar from jdbc.postgresql.org download page, people will find it > regardless of where the development happens. This sounds like the right approach to me. We need to ensure Postgres works with everything and everybody. We do want to help Blowfish, but not at an expense for other app servers. Sun chose to use Blowfish, but could have chosen others for the benchmark. We shouldn't go down a route with the software that is driven because of an external, unrelated decision. -- Simon Riggs 2ndQuadrant http://www.2ndQuadrant.com
On Fri, 2007-09-07 at 10:20 +0100, Heikki Linnakangas wrote: > Simon Riggs wrote: > > Sun chose to use Blowfish, but could have chosen others for the > > benchmark. We shouldn't go down a route with the software that is driven > > because of an external, unrelated decision. > > Blowfish? Is that a nickname for Sun Java System Application Server? Sorry, Glassfish. -- Simon Riggs 2ndQuadrant http://www.2ndQuadrant.com
On 7-Sep-07, at 5:17 AM, Simon Riggs wrote: > On Wed, 2007-09-05 at 21:20 +0100, Heikki Linnakangas wrote: >> Kris Jurka wrote: >>> On Wed, 5 Sep 2007, Dave Cramer wrote: >>>> If this is the case then I would argue that having caching in the >>>> appserver is a great idea for everyone using an appserver it >>>> does not >>>> help the rest of the world that doesn't use an app server. >>> >>> This is the consensus I'd like to see built before we spend a lot of >>> time on something. Heikki doesn't believe we should do this at >>> all as >>> it is covered by DBCP and app server pools. >> >> To be clear, I think it's nice to have one more statement cache >> implementation around, with friendly BSD license. I encourage you to >> keep working on it, regardless of how this discussion ends. >> >> But let's not bundle it with the PostgreSQL JDBC driver, it's >> clearly a >> separate component. For applications that already have another >> connection pool / statement cache implementation, it would be just >> extra >> baggage. And as a separate project, you can use it with other >> DBMSs as >> well. Or bundle it with an app server :). >> >> Another point of view is to think about the release cycles of the >> JDBC >> driver and the caching wrapper. If a bug is found in the wrapper, and >> it's bundled with the JDBC driver, we would have to release a new >> version of the bundle, forcing an upgrade of both components even >> if you >> only use one. If we add a new feature to the wrapper, is that >> going to >> be backpatched to say the 8.1 branch? What if you're running 8.1 >> server >> and driver, and you want to take advantage of a new wrapper feature? >> >> The wrapper has enough merit to live a happy and successful life as a >> project of its own. I don't care if it's a separate pgfoundry >> project, >> or just a separate CVS directory within jdbc project and separate >> builds; the main point is that it should be a separate jar file >> with a >> separate release cycle. If we add a link to the latest version of the >> wrapper jar from jdbc.postgresql.org download page, people will >> find it >> regardless of where the development happens. > > This sounds like the right approach to me. > > We need to ensure Postgres works with everything and everybody. We do > want to help Blowfish, but not at an expense for other app servers. How does the architecture of the driver hinder other app servers ? > > Sun chose to use Blowfish, but could have chosen others for the > benchmark. We shouldn't go down a route with the software that is > driven > because of an external, unrelated decision. After a quick survey I couldn't find another non-GPL open source app server. Dave
Dave Cramer wrote: > After a quick survey I couldn't find another non-GPL open source app > server. JOnAS and Apache Geronimo spring to mind. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com
On 7-Sep-07, at 7:33 AM, Heikki Linnakangas wrote: > Dave Cramer wrote: >> After a quick survey I couldn't find another non-GPL open source app >> server. > > JOnAS and Apache Geronimo spring to mind. JOnAS is LGPL > > -- > Heikki Linnakangas > EnterpriseDB http://www.enterprisedb.com
Dave Cramer wrote: > After a quick survey I couldn't find another non-GPL open source app > server. Isn't that a fairly arbitary categorization to make? How about open source app servers that don't begin with 'B'? ;-) Seriously, not sure exactly what point you're trying to make here. Why exactly should the existance and licensing of 3rd party software affect technical decisions about the postgresql JDBC driver? I think we're drifting away from the main point which is, as I see it, fairly simple: What is the implementation advantage of making statement pooling part of the main driver? There are maintenance issues which count *against* it being part of the driver so you need to provide a good reason to include it. Packaging issues are IMO fairly irrelevant because if you must give a single package to your users, you can always take the base driver and a separate pooling wrapper, package them together, and say "Hey look here is the statement-pooling postgresql JDBC driver". -O
On 7-Sep-07, at 9:13 AM, Oliver Jowett wrote: > Dave Cramer wrote: > >> After a quick survey I couldn't find another non-GPL open source >> app server. > > Isn't that a fairly arbitary categorization to make? How about open > source app servers that don't begin with 'B'? ;-) It's not arbitrary at all. see below > > Seriously, not sure exactly what point you're trying to make here. > Why exactly should the existance and licensing of 3rd party > software affect technical decisions about the postgresql JDBC driver? > Companies are fairly careful when choosing components to put in their stack. GPL licensed products are considered to be business non- friendly. I was responding to Simon's suggestion that they could have used any other app server which did implement statement caching. > I think we're drifting away from the main point which is, as I see > it, fairly simple: > > What is the implementation advantage of making statement pooling > part of the main driver? There are maintenance issues which count > *against* it being part of the driver so you need to provide a good > reason to include it. > Well, it has to be maintained regardless of where it is. So how does that make any difference ? > Packaging issues are IMO fairly irrelevant because if you must give > a single package to your users, you can always take the base driver > and a separate pooling wrapper, package them together, and say "Hey > look here is the statement-pooling postgresql JDBC driver". Dave > > -O
Sorry, I wanted to send this to the list...
---------- Forwarded message ----------
From: László Hornyák <kocka.public@gmail.com >
Date: 07-Sep-2007 15:55
Subject: Re: [JDBC] Caching driver on pgFoundry?
To: Oliver Jowett <oliver@opencloud.com>
Hi!
--
László Hornyák
--
László Hornyák
---------- Forwarded message ----------
From: László Hornyák <kocka.public@gmail.com >
Date: 07-Sep-2007 15:55
Subject: Re: [JDBC] Caching driver on pgFoundry?
To: Oliver Jowett <oliver@opencloud.com>
Hi!
On 07/09/07, Oliver Jowett <oliver@opencloud.com > wrote:
There are two main reasons I can mention:
- JDBC 4 allows prepared statement pooling (Which is optional, the driver can ignore the programs request to cache the prepared statement, and I believe it enables some kind of early optimization for those who like to do early optimizations. But that is just my opinion, but JDBC 4 specification expects drivers to deal with it. Time will tell, maybe data access layers like hibernate or JPA implementations in the future will use it) see http://java.sun.com/javase/6/docs/api/java/sql/Statement.html#setPoolable%28boolean%29
- Clients that do not use DataSource, but only DriverManager can gain significant speed improvement. Now, you get prepapred statement caching if the wrapper ( e.g. DBCP) gives it, not if you use a pg connection directly.
Yes, any developer can do that, but it would probably save some time for developers if they wouldn't have to.
What is the implementation advantage of making statement pooling part of
the main driver? There are maintenance issues which count *against* it
being part of the driver so you need to provide a good reason to include it.
There are two main reasons I can mention:
- JDBC 4 allows prepared statement pooling (Which is optional, the driver can ignore the programs request to cache the prepared statement, and I believe it enables some kind of early optimization for those who like to do early optimizations. But that is just my opinion, but JDBC 4 specification expects drivers to deal with it. Time will tell, maybe data access layers like hibernate or JPA implementations in the future will use it) see http://java.sun.com/javase/6/docs/api/java/sql/Statement.html#setPoolable%28boolean%29
- Clients that do not use DataSource, but only DriverManager can gain significant speed improvement. Now, you get prepapred statement caching if the wrapper ( e.g. DBCP) gives it, not if you use a pg connection directly.
Packaging issues are IMO fairly irrelevant because if you must give a
single package to your users, you can always take the base driver and a
separate pooling wrapper, package them together, and say "Hey look here
is the statement-pooling postgresql JDBC driver".
Yes, any developer can do that, but it would probably save some time for developers if they wouldn't have to.
-O
---------------------------(end of broadcast)---------------------------
TIP 9: In versions below 8.0, the planner will ignore your desire to
choose an index scan if your joining column's datatypes do not
match
--
László Hornyák
--
László Hornyák
László Hornyák wrote: > On 07/09/07, Oliver Jowett <oliver@opencloud.com> wrote: >> What is the implementation advantage of making statement pooling part of >> the main driver? There are maintenance issues which count *against* it >> being part of the driver so you need to provide a good reason to include >> it. > > There are two main reasons I can mention: > - JDBC 4 allows prepared statement pooling (Which is optional, the driver > can ignore the programs request to cache the prepared statement, and I > believe it enables some kind of early optimization for those who like to do > early optimizations. But that is just my opinion, but JDBC 4 specification > expects drivers to deal with it. Time will tell, maybe data access layers > like hibernate or JPA implementations in the future will use it) see > http://java.sun.com/javase/6/docs/api/java/sql/Statement.html#setPoolable%28boolean%29 As far as I can see, there's nothing in the spec that you can't implement in a generic wrapper. The methods added in JDBC4 allow the application to explicitly ask the connection pool/statement cache to not cache a statement. Implementation of that feature is exactly the same, whether the statement cache comes with the driver, the application server, or as a separate module. > - Clients that do not use DataSource, but only DriverManager can gain > significant speed improvement. Now, you get prepapred statement caching if > the wrapper ( e.g. DBCP) gives it, not if you use a pg connection directly. Nothing stops you from using the caching wrapper with the DriverManager interface. I believe DBCP supports that as well. > Packaging issues are IMO fairly irrelevant because if you must give a >> single package to your users, you can always take the base driver and a >> separate pooling wrapper, package them together, and say "Hey look here >> is the statement-pooling postgresql JDBC driver". > > Yes, any developer can do that, but it would probably save some time for > developers if they wouldn't have to. Frankly I don't understand what the benefit of a combined jar is. Surely if you can add one jar to your classpath, you can add two. But I won't object if someone wants to maintain such a combined jar; it's not going to harm me or anyone else that such a thing exists, as long as you're not forced to use it. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com
People, > What is the implementation advantage of making statement pooling part of > the main driver? There are maintenance issues which count *against* it > being part of the driver so you need to provide a good reason to include > it. How about if we wait for the performance numbers before settling this? -- Josh Berkus PostgreSQL @ Sun San Francisco
Dave Cramer wrote: > > On 7-Sep-07, at 9:13 AM, Oliver Jowett wrote: >> >> What is the implementation advantage of making statement pooling part >> of the main driver? There are maintenance issues which count *against* >> it being part of the driver so you need to provide a good reason to >> include it. >> > Well, it has to be maintained regardless of where it is. So how does > that make any difference ? Err if it is part of the main driver it needs to be maintained as part of the main driver and will follow the main driver's release cycle. Others have commented on the same thing. It also increases the complexity of the main driver, making it harder for a particular person to maintain the whole thing. I'm going to ask again: What is the implementation advantage of making statement pooling part of the main driver? -O
Josh Berkus wrote: > People, > >> What is the implementation advantage of making statement pooling part of >> the main driver? There are maintenance issues which count *against* it >> being part of the driver so you need to provide a good reason to include >> it. > > How about if we wait for the performance numbers before settling this? Sure, if they're on their way that's fine, Dave hasn't responded to my earier email asking about it. -O
On 8-Sep-07, at 5:38 PM, Oliver Jowett wrote: > Dave Cramer wrote: >> On 7-Sep-07, at 9:13 AM, Oliver Jowett wrote: >>> >>> What is the implementation advantage of making statement pooling >>> part of the main driver? There are maintenance issues which count >>> *against* it being part of the driver so you need to provide a >>> good reason to include it. >>> >> Well, it has to be maintained regardless of where it is. So how >> does that make any difference ? > > Err if it is part of the main driver it needs to be maintained as > part of the main driver and will follow the main driver's release > cycle. Others have commented on the same thing. It also increases > the complexity of the main driver, making it harder for a > particular person to maintain the whole thing. > > I'm going to ask again: What is the implementation advantage of > making statement pooling part of the main driver? > It's simpler, there's about half a dozen classes and one if statement as opposed to implementing the Connection and Statement interfaces for 3 versions of jdbc and yet another build system that does the same. Dave > -O
Dave Cramer wrote: > It's simpler, there's about half a dozen classes and one if statement as > opposed to implementing the Connection and Statement interfaces for 3 > versions of jdbc and yet another build system that does the same. How do you avoid wrapping Connection/Statement/etc and still avoid the problems we discussed with Statement.close()? Can you send me a patch for review? -O
Hi Oliver, I'm working on the other wrapper driver now. I want to finish *something* and get it working before I start looking at the issues that you raised. As you are eluding to it might just be as complicated to get it right in the driver, but at this point I have concentrate on the wrapper implementation. Dave On 8-Sep-07, at 6:57 PM, Oliver Jowett wrote: > Dave Cramer wrote: > >> It's simpler, there's about half a dozen classes and one if >> statement as opposed to implementing the Connection and Statement >> interfaces for 3 versions of jdbc and yet another build system >> that does the same. > > How do you avoid wrapping Connection/Statement/etc and still avoid > the problems we discussed with Statement.close()? Can you send me a > patch for review? > > -O
Dave Cramer wrote: > Hi Oliver, > > I'm working on the other wrapper driver now. I want to finish > *something* and get it working before I start looking at the issues that > you raised. As you are eluding to it might just be as complicated to get > it right in the driver, but at this point I have concentrate on the > wrapper implementation. Ok, how about we shelve this discussion until you're done with the wrapper, and looked at how you'd do something equivalent as part of the driver, and got some rough comparisons between the two? -O
Dave Cramer wrote: > It's simpler, there's about half a dozen classes and one if statement as > opposed to implementing the Connection and Statement interfaces for 3 > versions of jdbc and yet another build system that does the same. I think it's reasonable to assume that the cache will only be used in new applications, so you don't need to support as many JDBC versions as the driver itself. Actually, do you really need to have a separate version for each JDBC version? Wouldn't a wrapper written and compiled with JDBC4 work with older JDKs as well, as long as the application doesn't use the JDBC4 features? I might be missing something, I don't fully understand how it works in the core driver either. Have you considered using java.lang.reflect.Proxy? It has some overhead, but is it significant in modern JVMs? -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com
> > On 7-Sep-07, at 9:13 AM, Oliver Jowett wrote: > > > Dave Cramer wrote: > > > >> After a quick survey I couldn't find another non-GPL open > source app > >> server. > > > > Isn't that a fairly arbitary categorization to make? How about open > > source app servers that don't begin with 'B'? ;-) > > It's not arbitrary at all. see below > > > > Seriously, not sure exactly what point you're trying to make here. > > Why exactly should the existance and licensing of 3rd party > software > > affect technical decisions about the postgresql JDBC driver? > > > Companies are fairly careful when choosing components to put > in their stack. GPL licensed products are considered to be > business non- friendly. I was responding to Simon's > suggestion that they could have used any other app server > which did implement statement caching. > As a "satisfied" postgresql jdbc user and someone who works with a couple of open source appservers I think its fair to point out that while GPL is considered by many to be "non business friendly", many businesses are happy with the LGPL licence. Jboss is released under the LGPL and the jboss connection pool does include statement caching (ie, the pool remembers which statements have been prepared against which connections - which is my understanding of the purpose of this thread). So, if you are looking for an open source java/jee with postgresql option that includes connection pool level caching of prepared statements then there is already at least one option out there. My own view is that I'd be suspicious of including connection level caching in the same package as the standard JDBC driver if: 1) the pooling code slows down the standard driver when the pooling isn't used 2) the release cycle of the pooling code "hobbles" the release of the main driver 3) the pooling code breaks existing external pooling/caching options Anyway, thats my £0.02 - thanks for all the good work :) Cheers, Andy > > I think we're drifting away from the main point which is, > as I see it, > > fairly simple: > > > > What is the implementation advantage of making statement > pooling part > > of the main driver? There are maintenance issues which count > > *against* it being part of the driver so you need to provide a good > > reason to include it. > > > Well, it has to be maintained regardless of where it is. So > how does that make any difference ? > > > Packaging issues are IMO fairly irrelevant because if you > must give a > > single package to your users, you can always take the base > driver and > > a separate pooling wrapper, package them together, and say > "Hey look > > here is the statement-pooling postgresql JDBC driver". > > Dave > > > > -O > > > ---------------------------(end of > broadcast)--------------------------- > TIP 5: don't forget to increase your free space map settings >