Обсуждение: Backend connection pooling memleak
Hello, I implemented connection pooling in the backend of postgresql (one of the TODO items) and I've found some really interesting performance improvements I'd like to continue to explore but unfortunately I've also uncovered a nasty little side effect with the memory usage. It's my guess that it's either a number of mallocs don't have corresponding frees because the process would exit when the transaction is done (normally) or I've bypassed some of the memory context "free"s that would have also resulted in memory being reclaimed on a process exit or more importantly on a connection being reclosed. The implementation is different from front-end pooling in that the client does not keep their connection open with the database. The socket is closed for example. Also a basic authorization check (to make sure the client credentials match those of an existing pooled connection) happens. So the connection does get some of the initialization that occurs for a new connection (but not all, a lot of it can be skipped because it has already been setup). Anyway, my hope is there might be someone who is familiar with either the memory system or the connection startup and shutdown process enough that could provide a few pointers as to where I could start looking. I've been instrumenting the code to track malloc/pmalloc and free/pfree calls to see which ones don't match for example but it's a bit laborious and would be simpler if I could narrow down where I think things might be having problems. The implementation was done on 7.3.4. It was the latest at the time I started and I haven't sync'd up yet because I wasn't sure if there would be a need in terms of the general postgresql population wanting this feature. If the improvements prove to be real and generalized versus specialized (just a few use cases for example) then I would imagine the interest would grow but I'm not there yet. I can get you copies of the changes I've made. I changed 27 files (some of the changes are minor, some more complicated) and I added 2 files as well (for the pooling functionality). Thank you, Mike
On Tue, Nov 01, 2005 at 11:33:39PM -0700, Mike Shelton wrote: > Hello, > > I implemented connection pooling in the backend of postgresql (one of the > TODO items) and I've found some really interesting performance improvements > I'd like to continue to explore but unfortunately I've also uncovered a > nasty little side effect with the memory usage. It's my guess that it's > either a number of mallocs don't have corresponding frees because the > process would exit when the transaction is done (normally) or I've bypassed > some of the memory context "free"s that would have also resulted in memory > being reclaimed on a process exit or more importantly on a connection being > reclosed. You should probably look at the memory management documentation. Many (most?) pmallocs are not explicitly freed. Instead, the context they are allocated in is either reset or freed. So in your case, it should simply be a matter of resetting some of the nearly top level contexts. See src/backend/utils/mmgr/README > The implementation was done on 7.3.4. It was the latest at the time I > started and I haven't sync'd up yet because I wasn't sure if there would be > a need in terms of the general postgresql population wanting this feature. > If the improvements prove to be real and generalized versus specialized > (just a few use cases for example) then I would imagine the interest would > grow but I'm not there yet. The question is, are the improvements worth the costs? What does your implementation do that something like pgpool doesn't? Good luck. Have a nice day, -- Martijn van Oosterhout <kleptog@svana.org> http://svana.org/kleptog/ > Patent. n. Genius is 5% inspiration and 95% perspiration. A patent is a > tool for doing 5% of the work and then sitting around waiting for someone > else to do the other 95% so you can sue them.