SSL encryption makes bytea transfer slow

We selected a 30MB bytea with psql connected with
"-h localhost" and found that it makes a huge
difference whether we have SSL encryption on or off.

Without SSL the SELECT finished in about a second,
with SSL it took over 23 seconds (measured with
\timing in psql).
During that time, the CPU is 100% busy.
All data are cached in memory.

Is this difference as expected?

Yours,
Laurenz Albe