libpq contention due to gss even when not using gss

Hi,

To investigate a report of both postgres and pgbouncer having issues when a
lot of new connections aree established, I used pgbench -C.  Oddly, on an
early attempt, the bottleneck wasn't postgres+pgbouncer, it was pgbench. But
only when using TCP, not with unix sockets.

c=40;pgbench -C -n -c$c -j$c -T5 -f <(echo 'select 1') 'port=6432 host=127.0.0.1 user=test dbname=postgres
password=fake'

host=127.0.0.1:                           16465
host=127.0.0.1,gssencmode=disable         20860
host=/tmp:                                49286

Note that the server does *not* support gss, yet gss has a substantial
performance impact.

Obviously the connection rates here absurdly high and outside of badly written
applications likely never practically relevant. However, the number of cores
in systems are going up, and this quite possibly will become relevant in more
realistic scenarios (lock contention kicks in earlier the more cores you
have).

And it doesn't seem great that something as rarely used as gss introduces
overhead to very common paths.

Here's a bottom-up profile:

-   32.10%  pgbench  [kernel.kallsyms]      [k] queued_spin_lock_slowpath
   - 32.09% queued_spin_lock_slowpath
      - 16.15% futex_wake
           do_futex
           __x64_sys_futex
           do_syscall_64
         - entry_SYSCALL_64_after_hwframe
            - 16.15% __GI___lll_lock_wake
               - __GI___pthread_mutex_unlock_usercnt
                  - 5.12% gssint_select_mech_type
                     - 4.36% gss_inquire_attrs_for_mech
                        - 2.85% gss_indicate_mechs
                           - gss_indicate_mechs_by_attrs
                              - 1.58% gss_acquire_cred_from
                                   gss_acquire_cred
                                   pg_GSS_have_cred_cache
                                   select_next_encryption_method (inlined)
                                   init_allowed_encryption_methods (inlined)
                                   PQconnectPoll
                                   pqConnectDBStart (inlined)
                                   PQconnectStartParams
                                   PQconnectdbParams
                                   doConnect


And a bottom-up profile:

-   32.10%  pgbench  [kernel.kallsyms]      [k] queued_spin_lock_slowpath
   - 32.09% queued_spin_lock_slowpath
      - 16.15% futex_wake
           do_futex
           __x64_sys_futex
           do_syscall_64
         - entry_SYSCALL_64_after_hwframe
            - 16.15% __GI___lll_lock_wake
               - __GI___pthread_mutex_unlock_usercnt
                  - 5.12% gssint_select_mech_type
                     - 4.36% gss_inquire_attrs_for_mech
                        - 2.85% gss_indicate_mechs
                           - gss_indicate_mechs_by_attrs
                              - 1.58% gss_acquire_cred_from
                                   gss_acquire_cred
                                   pg_GSS_have_cred_cache
                                   select_next_encryption_method (inlined)
                                   init_allowed_encryption_methods (inlined)
                                   PQconnectPoll
                                   pqConnectDBStart (inlined)
                                   PQconnectStartParams
                                   PQconnectdbParams
                                   doConnect



Clearly the contention originates outside of our code, but is triggered by
doing pg_GSS_have_cred_cache() every time a connection is established.

Greetings,

Andres Freund