Обсуждение: accumulating handles problem on machine running postgresql
I’m running Postgresql 9.0.3 on a Windows 7 box. There is one large database that gets inserted to thousands of times a day every day. The problem is the Handle count on the Windows System process of the server, image name c:\windows\system32\ntoskrnl.exe, accumulates over time and the handles never get released. This causes the handle count to go to about 130,000 at which time the machine starts malfunctioning and a reboot is needed. It takes about 6 days before a reboot is needed. This is causing our regression test machines, which do inserts into the database via a gsoap connection to a web service running on the database machine, to get bungled up every 6 days or so. Has anyone had a problem with their postgresql server where the System process accumulates handles and doesn’t release them?
P.S. In addition to postgresql, the Windows 7 server is running IIS 7 and asp.net web services. The web services are reached via gSOAP from various test machines on the local network, some running windows and some running linux.
I really wish I knew what was causing this. I guess it could be postgresql, the devart bindings, gsoap or the web service code itself. I’m sure I’m closing all database connections in the web service code. The database and web services function perfectly except for the handle problem.
Thanks,
Adam
Adam Bruss
Senior Development Engineer
AWR Corporation
11520 N. Port Washington Rd., Suite 201
Mequon, WI 53092 USA
P: 1.262.240.0291 x104
F: 1.262.240.0294
On 02/27/12 6:34 PM, Adam Bruss wrote: > > I’m running Postgresql 9.0.3 on a Windows 7 box. There is one large > database that gets inserted to thousands of times a day every day. The > problem is the Handle count on the Windows System process of the > server, image name c:\windows\system32\ntoskrnl.exe, accumulates over > time and the handles never get released. This causes the handle count > to go to about 130,000 at which time the machine starts malfunctioning > and a reboot is needed. It takes about 6 days before a reboot is > needed. This is causing our regression test machines, which do inserts > into the database via a gsoap connection to a web service running on > the database machine, to get bungled up every 6 days or so. Has anyone > had a problem with their postgresql server where the System process > accumulates handles and doesn’t release them? > > P.S. In addition to postgresql, the Windows 7 server is running IIS 7 > and asp.net web services. The web services are reached via gSOAP from > various test machines on the local network, some running windows and > some running linux. > > I really wish I knew what was causing this. I guess it could be > postgresql, the devart bindings, gsoap or the web service code itself. > I’m sure I’m closing all database connections in the web service code. > The database and web services function perfectly except for the handle > problem. > IIS7 runs the websites half in the kernel, which is the 'system' process... I'd be suspicious of something happening over there, and not in the postgres database server. you want to try a test? run postgres on a different system, and have your webpile connect to it over the LAN. if you're still loosing handles, its definitely not PG's fault. -- john r pierce N 37, W 122 santa cruz ca mid-left coast
On Tue, Feb 28, 2012 at 04:34, Adam Bruss <abruss@awrcorp.com> wrote: > The problem is the > Handle count on the Windows System process of the server, image name > c:\windows\system32\ntoskrnl.exe, accumulates over time and the handles > never get released. This causes the handle count to go to about 130,000 at > which time the machine starts malfunctioning and a reboot is needed. Do these handles go away when you restart any of these services? If it's a user space code problem then the handles should get freed automatically after the process exits. Regards, Marti
The handles persist through restarting the postgresql service and restarting the IIS server. The handles are accumulatingon the System process. I think the handles are created when the web service is accessed but that would mean theIIS worker processes would have responsibility and they don't seem to. Recycling the worker processes in IIS does nothing.And the worker processes have their own process w3wp.exe which never accumulate handles. It's probably not a postgresqlthing or other people would be seeing it. Adam Bruss Senior Development Engineer AWR Corporation 11520 N. Port Washington Rd., Suite 201 Mequon, WI 53092 USA P: 1.262.240.0291 x104 F: 1.262.240.0294 E: abruss@awrcorp.com W: http://www.awrcorp.com -----Original Message----- From: Marti Raudsepp [mailto:marti@juffo.org] Sent: Tuesday, February 28, 2012 8:23 AM To: Adam Bruss Cc: pgsql-general@postgresql.org Subject: Re: [GENERAL] accumulating handles problem on machine running postgresql On Tue, Feb 28, 2012 at 04:34, Adam Bruss <abruss@awrcorp.com> wrote: > The problem is the > Handle count on the Windows System process of the server, image name > c:\windows\system32\ntoskrnl.exe, accumulates over time and the handles > never get released. This causes the handle count to go to about 130,000 at > which time the machine starts malfunctioning and a reboot is needed. Do these handles go away when you restart any of these services? If it's a user space code problem then the handles should get freed automatically after the process exits. Regards, Marti
On Tue, Feb 28, 2012 at 8:48 AM, Adam Bruss <abruss@awrcorp.com> wrote: > The handles persist through restarting the postgresql service and restarting the IIS server. The handles are accumulatingon the System process. I think the handles are created when the web service is accessed but that would mean theIIS worker processes would have responsibility and they don't seem to. Recycling the worker processes in IIS does nothing.And the worker processes have their own process w3wp.exe which never accumulate handles. It's probably not a postgresqlthing or other people would be seeing it. > Use "process explorer" from sysinternals / microsoft (google for it) to see what these handles are for (pipes, files, events, mutants, desktops, winstations (ok, probably not those), etc...
I ran process explorer and looked at the handles for the System process. The vast majority of the handles are of type
"Key".I can find them in the registry. I took two at random from process explorer and exported the registry branch for
thembelow.
## EXAMPLE 1: ##
Key Name: HKEY_CLASSES_ROOT\CLSID\{9F074EE2-E6E9-4d8a-A047-EB5B5C3C55DA}
Class Name: <NO CLASS>
Last Write Time: 2/28/2012 - 1:26 AM
Value 0
Name: <NO NAME>
Type: REG_SZ
Data: HwTextInsertion Class
Key Name: HKEY_CLASSES_ROOT\CLSID\{9F074EE2-E6E9-4d8a-A047-EB5B5C3C55DA}\InprocServer32
Class Name: <NO CLASS>
Last Write Time: 2/29/2012 - 4:05 AM
Value 0
Name: <NO NAME>
Type: REG_EXPAND_SZ
Data: %CommonProgramFiles%\microsoft shared\ink\tiptsf.dll
Value 1
Name: ThreadingModel
Type: REG_SZ
Data: Apartment
Key Name: HKEY_CLASSES_ROOT\CLSID\{9F074EE2-E6E9-4d8a-A047-EB5B5C3C55DA}\ProgID
Class Name: <NO CLASS>
Last Write Time: 2/29/2012 - 4:05 AM
Key Name: HKEY_CLASSES_ROOT\CLSID\{9F074EE2-E6E9-4d8a-A047-EB5B5C3C55DA}\Server
Class Name: <NO CLASS>
Last Write Time: 2/29/2012 - 4:05 AM
## EXAMPLE 2: ##
Key Name: HKEY_LOCAL_MACHINE\SOFTWARE\Classes\CLSID\{80FF6842-51A9-4959-B3B9-EE4DCBFD7740}
Class Name: <NO CLASS>
Last Write Time: 2/28/2012 - 12:07 AM
Key Name: HKEY_LOCAL_MACHINE\SOFTWARE\Classes\CLSID\{80FF6842-51A9-4959-B3B9-EE4DCBFD7740}\Programmable
Class Name: <NO CLASS>
Last Write Time: 12/13/2010 - 12:27 PM
Key Name: HKEY_LOCAL_MACHINE\SOFTWARE\Classes\CLSID\{80FF6842-51A9-4959-B3B9-EE4DCBFD7740}\InprocServer32
Class Name: <NO CLASS>
Last Write Time: 2/29/2012 - 3:05 AM
Key Name: HKEY_LOCAL_MACHINE\SOFTWARE\Classes\CLSID\{80FF6842-51A9-4959-B3B9-EE4DCBFD7740}\ProgID
Class Name: <NO CLASS>
Last Write Time: 2/29/2012 - 3:05 AM
Key Name: HKEY_LOCAL_MACHINE\SOFTWARE\Classes\CLSID\{80FF6842-51A9-4959-B3B9-EE4DCBFD7740}\Server
Class Name: <NO CLASS>
Last Write Time: 2/29/2012 - 3:05 AM
## END EXAMPLES ##
A common thread I notice when looking through the keys is InprocServer32.
Adam Bruss
Senior Development Engineer
AWR Corporation
11520 N. Port Washington Rd., Suite 201
Mequon, WI 53092 USA
P: 1.262.240.0291 x104
F: 1.262.240.0294
E: abruss@awrcorp.com
W: http://www.awrcorp.com
-----Original Message-----
From: pgsql-general-owner@postgresql.org [mailto:pgsql-general-owner@postgresql.org] On Behalf Of dennis jenkins
Sent: Tuesday, February 28, 2012 4:01 PM
To: pgsql-general@postgresql.org
Subject: Re: [GENERAL] accumulating handles problem on machine running postgresql
On Tue, Feb 28, 2012 at 8:48 AM, Adam Bruss <abruss@awrcorp.com> wrote:
> The handles persist through restarting the postgresql service and restarting the IIS server. The handles are
accumulatingon the System process. I think the handles are created when the web service is accessed but that would mean
theIIS worker processes would have responsibility and they don't seem to. Recycling the worker processes in IIS does
nothing.And the worker processes have their own process w3wp.exe which never accumulate handles. It's probably not a
postgresqlthing or other people would be seeing it.
>
Use "process explorer" from sysinternals / microsoft (google for it)
to see what these handles are for (pipes, files, events, mutants,
desktops, winstations (ok, probably not those), etc...
--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general
On Wed, Feb 29, 2012 at 8:54 AM, Adam Bruss <abruss@awrcorp.com> wrote:
> I ran process explorer and looked at the handles for the System process. The vast majority of the handles are of type
"Key".I can find them in the registry. I took two at random from process explorer and exported the registry branch for
thembelow.
>
> ## EXAMPLE 1: ##
>
> Key Name: HKEY_CLASSES_ROOT\CLSID\{9F074EE2-E6E9-4d8a-A047-EB5B5C3C55DA}
> Class Name: <NO CLASS>
> Last Write Time: 2/28/2012 - 1:26 AM
> Value 0
> Name: <NO NAME>
> Type: REG_SZ
> Data: HwTextInsertion Class
>
>
> Key Name: HKEY_CLASSES_ROOT\CLSID\{9F074EE2-E6E9-4d8a-A047-EB5B5C3C55DA}\InprocServer32
> Class Name: <NO CLASS>
> Last Write Time: 2/29/2012 - 4:05 AM
> Value 0
> Name: <NO NAME>
> Type: REG_EXPAND_SZ
> Data: %CommonProgramFiles%\microsoft shared\ink\tiptsf.dll
>
> Value 1
> Name: ThreadingModel
> Type: REG_SZ
> Data: Apartment
Seems like your web server is leaking registry keys used when loading
COM objects. The sample that you posted is for the "Tablet PC Input
Panel Text Services Framework" [1]. However, I find it strange that
a) IIS needs this and b) that it would leak it.
Are you able to obtain a large statistical sample of the leaked
registry keys? 2 out of 130,000 seems like a small sample.
Try the command line "handle.exe" tool [2]. It can dump to a text
file that you can then analyze with perl, python, grep, etc... or your
own eyeballs. :) See if the handle list is dominated by a specific
set of registry keys.
[1] http://systemexplorer.net/filereviews.php?fid=515344
[2] http://technet.microsoft.com/en-us/sysinternals/bb896655
After doing some more work with Process Monitor I found the leaks to be stemming from a driver associated with a LicenseManager from SafeNet Inc. which runs on the machine. Adam Bruss Senior Development Engineer AWR Corporation 11520 N. Port Washington Rd., Suite 201 Mequon, WI 53092 USA P: 1.262.240.0291 x104 F: 1.262.240.0294 E: abruss@awrcorp.com W: http://www.awrcorp.com -----Original Message----- From: dennis jenkins [mailto:dennis.jenkins.75@gmail.com] Sent: Wednesday, February 29, 2012 12:25 PM To: Adam Bruss Cc: pgsql-general@postgresql.org Subject: Re: [GENERAL] accumulating handles problem on machine running postgresql On Wed, Feb 29, 2012 at 8:54 AM, Adam Bruss <abruss@awrcorp.com> wrote: > I ran process explorer and looked at the handles for the System process. The vast majority of the handles are of type "Key".I can find them in the registry. I took two at random from process explorer and exported the registry branch for thembelow. > > ## EXAMPLE 1: ## > > Key Name: HKEY_CLASSES_ROOT\CLSID\{9F074EE2-E6E9-4d8a-A047-EB5B5C3C55DA} > Class Name: <NO CLASS> > Last Write Time: 2/28/2012 - 1:26 AM > Value 0 > Name: <NO NAME> > Type: REG_SZ > Data: HwTextInsertion Class > > > Key Name: HKEY_CLASSES_ROOT\CLSID\{9F074EE2-E6E9-4d8a-A047-EB5B5C3C55DA}\InprocServer32 > Class Name: <NO CLASS> > Last Write Time: 2/29/2012 - 4:05 AM > Value 0 > Name: <NO NAME> > Type: REG_EXPAND_SZ > Data: %CommonProgramFiles%\microsoft shared\ink\tiptsf.dll > > Value 1 > Name: ThreadingModel > Type: REG_SZ > Data: Apartment Seems like your web server is leaking registry keys used when loading COM objects. The sample that you posted is for the "Tablet PC Input Panel Text Services Framework" [1]. However, I find it strange that a) IIS needs this and b) that it would leak it. Are you able to obtain a large statistical sample of the leaked registry keys? 2 out of 130,000 seems like a small sample. Try the command line "handle.exe" tool [2]. It can dump to a text file that you can then analyze with perl, python, grep, etc... or your own eyeballs. :) See if the handle list is dominated by a specific set of registry keys. [1] http://systemexplorer.net/filereviews.php?fid=515344 [2] http://technet.microsoft.com/en-us/sysinternals/bb896655