The ssl connection is slower, and it is expected. But when I configure pg_hba.conf to disable ssl via hostnossl, then ssl is not used, but the speed is similar to ssl.
Is it expected behave?
That's definitely not expected behavior. hostnossl should turn off ssl which should turn off the overhead completely. Does it make a difference if you also disable it from the client side?
When I explicitly disabled ssl, then I seen significantly less time
Intersting. Can you check with a network trace that it actually turns off ssl, so nothing is broken there?
One thing that could be taking the time is an extra roundtrip -- e.g. it tries to connect with ssl fails and retries without. A network trace should also make this obvious, and can hopefully show you exactly where in the connection the time is spent.