Auto Vacuum of pg_catalog tables causes huge delay in opening new connections

   We have a interesting thing happening on one of our DB's that when autovacuum runs against the pg_catalog tables (or we run a manual vacuum) we get a large delay in opening new connections.  If this happens during a busy period on our system we end up getting 200-300 connections trying to open at once.  All the backend processes on the server for the connections show up in the ps list in a state of startup.  As the connections don't actually open until the delay clears the maximum connections in the connection pool we have configured on our application servers is not getting hit causing the large number of connections to be attempted open.   The performance of queries on the existing open connections are also effected (which is why more connections are being opened since the existing open ones are releasing back to the pool slower).

The database contains roughly 1300 schemas each with about 1350 objects. Each schema is used to store the data for a single client.  As part of our demoing system we create 100 new schemas each day.  And have the same number of schemas dropped each day due to expired demos.

On a test server when I manually run a vacuum on pg_catalog.pg_class and then attempt to open 20 connections from a remote client to the DB I get an average of 64 seconds to get the connections open with the longest taking 80 seconds

If I immediately run another vacuum of pg_class (with no drop or creation of any schemas) and then attempt to open 20 connections from the same client I get an average of 84 seconds to get the connections open with the longest taking 123 seconds.  The behavior seems to not be 100% consistent however as some times the subsequent vacuum run doesn't cause the connections to be slow.  

We pulled a core dump from one of the backend processes when it was in this state and got the following backtrace

As a stop gap we've added 3 additional schema storage DB's and are rotating the creations between them to reduce the number of schemas in the single DB that we had (also reducing the removal and addition of schemas to that DB).  To reduce the impact of the vacuum on what used to be our single DB.  But this is only gaining us some time as if the vacuum hits us at a busy time for requests we will get the same backup of 200-300 connections in the startup state.

Any one have any suggestions as to how to resolve this? 

The documentation suggests that running the auto vacuum more often can help with performance issues but with the results I'm seeing on our test box it seems like that doing that is just going to cause more issues not less.

Thanks

--Mike