Обсуждение: Compression on SSL links?
I may be blind - I don't see a way to enable this. OpenSSL "kinda" supports this - does Postgres' SSL connectivity allow it to be supported/enabled? - Karl
Вложения
Karl Denninger wrote: > I may be blind - I don't see a way to enable this. OpenSSL "kinda" > supports this - does Postgres' SSL connectivity allow it to be > supported/enabled? What are you asking, exactly? -- Bruce Momjian <bruce@momjian.us> http://momjian.us EnterpriseDB http://enterprisedb.com + It's impossible for everything to be true. +
On 13/08/2010 9:31 PM, Bruce Momjian wrote: > Karl Denninger wrote: >> I may be blind - I don't see a way to enable this. OpenSSL "kinda" >> supports this - does Postgres' SSL connectivity allow it to be >> supported/enabled? > > What are you asking, exactly? As far as I can tell they're asking for transport-level compression, using gzip or similar, in much the same way as SSL/TLS currently provides transport-level encryption. Compression at the postgresql protocol level or above, so it's invisible at the level of the libpq APIs for executing statements and processing results, and doesn't change SQL processing. Since remote access is often combined with SSL, which is already supported by libpq, using SSL-integrated compression seems pretty promising if it's viable in practice. It'd avoid the pain of having to add compression to the Pg protocol by putting it "outside" the current protocol, in the SSL layer. Even better, compressing results before encrypting them makes the encrypted traffic *much* stronger against known-plaintext and pattern-based attacks. And, of course, compressing the content costs CPU time but reduces the amount of data that must then be compressed. OpenSSL does provide some transparent crypto support. See: http://www.openssl.org/docs/ssl/SSL_COMP_add_compression_method.html That page notes that using compression as integrated into the SSL layer has a few issues. The lack of compression method standardization shouldn't be a problem for PostgreSQL, as PostgreSQL is free to define compression methods and their names as an SSL-only extension to the current postgresql-specific protocol. It looks like the negotiation mechanisms to make this transparent to older clients are there, too, so there's potential, though lots of testing would be required. It's hard to know how widely the COMP_zlib method is enabled in deployed copies of OpenSSL, but a configure test could probably detect its availibility and conditionally enable it in libpq. Again, connections would just fall back to uncompressed. I'm curious to see if the Java SSL APIs can support SSL-level compression, and if so whether the JDBC driver can be SSL+Gzip enabled. I'll have a dig through the JCA (Java Cryptography Architecture) docs and see what I can find out. -- Craig Ringer
Craig Ringer wrote: > On 13/08/2010 9:31 PM, Bruce Momjian wrote: > > Karl Denninger wrote: > >> I may be blind - I don't see a way to enable this. OpenSSL "kinda" > >> supports this - does Postgres' SSL connectivity allow it to be > >> supported/enabled? > > > > What are you asking, exactly? > > As far as I can tell they're asking for transport-level compression, > using gzip or similar, in much the same way as SSL/TLS currently > provides transport-level encryption. Compression at the postgresql > protocol level or above, so it's invisible at the level of the libpq > APIs for executing statements and processing results, and doesn't change > SQL processing. > > Since remote access is often combined with SSL, which is already > supported by libpq, using SSL-integrated compression seems pretty > promising if it's viable in practice. It'd avoid the pain of having to > add compression to the Pg protocol by putting it "outside" the current > protocol, in the SSL layer. Even better, compressing results before > encrypting them makes the encrypted traffic *much* stronger against > known-plaintext and pattern-based attacks. And, of course, compressing > the content costs CPU time but reduces the amount of data that must then > be compressed. > > OpenSSL does provide some transparent crypto support. See: > http://www.openssl.org/docs/ssl/SSL_COMP_add_compression_method.html I thought all SSL traffic was compressed, unless you turned that off. It is just SSH that is always compressed? -- Bruce Momjian <bruce@momjian.us> http://momjian.us EnterpriseDB http://enterprisedb.com + It's impossible for everything to be true. +
On 13/08/2010 10:43 PM, Craig Ringer wrote: > OpenSSL does provide some transparent crypto support. See: > http://www.openssl.org/docs/ssl/SSL_COMP_add_compression_method.html Some more info on this: Apache mod_ssl docs mention that SSLv3 handshake is required to negotiate compression in SSL. That shouldn't be a big issue for postgresql, and at worst would just require a flag to switch back to SSLv2 for incredibly ancient servers (if there are even any out there). http://httpd.apache.org/docs/2.0/ssl/ssl_faq.html#comp It also shows that OpenSSL's docs are out of date, because there's a standard way to negotiate DEFLATE compression (zlib/gzip). Not that Pg would need that, but it's nice to have a standard name to use and it might help with non-OpenSSL implementations like JSSE/JCA for Java/JDBC. See: http://www.ietf.org/rfc/rfc3749.txt Unfortunately there's no reference to DEFLATE or to compression in the JSSE standard docs, and the latest information I can find (admittedly from 2007) says "Compression is not supported by Sun's JSSE provider. ". Sigh. No compression for PgJDBC SSL. There's a 3rd pty JSSE provider called "Jessie" that claims to support SSL compression, but it looks pretty dead and I don't see people wanting compression that badly anyway. So: It looks like SSL/TLS level compression is supported by OpenSSL, but I don't see much indication that it's widely adopted. I can't find any evidence Microsoft support it in their SSL/TLS implementation either. Win2k3's explicitly lacks it, so even if it's been added since it won't be widely deployed. Overall: it sounds to me like SSL/TLS level compression would only be useful for native libpq-to-postgresql connections, and probably wouldn't be usable for non-libpq based database access drivers. It'd only work if SSL was configured, which is a limitation it'd be nice to avoid. Plus, it doesn't look like it's a clear-cut obviously backward compatible change - there's the need to use the SSLv3 handshake, issues with late standardization of compression protocol names, the fact that distributions of OpenSSL might not be built with compression support, etc. Personally, after looking into it a little, I'm not even close to interested enough to do the required patching and compatibility testing. Adding compression to the postgresql protocol would be much harder, but looks like it'd be much more worthwhile. It's not something I personally care about or want to look into though. -- Craig Ringer
On 13/08/2010 10:50 PM, Bruce Momjian wrote: > I thought all SSL traffic was compressed, unless you turned that off. > It is just SSH that is always compressed? Frankly, I thought all SSL traffic was compressed too, but the reading I've just been doing suggests otherwise. It looks like compression *is* done as part of cyphering and hashing and is used during key exchange etc, but the actual application data being transported isn't compressed. At least, that's how it seems to me from the digging I've just been doing, though I want to look into it more. AFAIK SSH isn't always compressed. It's only compressed if you pass the "-C" flag, set -o Compression=yes, or add Compression=yes to .ssh/config or /etc/ssh/ssh_config . I'll admit being surprised, as it's widely stated that crypto is much stronger if the data to be protected is compressed first. I guess the CPU costs are significant enough that it's not widely done for bulk data, though, only for the critical parts like negotiating and exchanging the session key. -- Craig Ringer
Craig Ringer <craig@postnewspapers.com.au> writes: > Overall: it sounds to me like SSL/TLS level compression would only be > useful for native libpq-to-postgresql connections, and probably wouldn't > be usable for non-libpq based database access drivers. It'd only work if > SSL was configured, which is a limitation it'd be nice to avoid. I don't really see that. The case where it's sensible to use compression on the connection is where you're pushing data across a WAN. That's also pretty much exactly the situation where it's sensible to use encryption. I guess there'd be a use case for compression-without-encryption if you had a trustworthy private WAN, but who's got one of those? So I'm much more in favor of doing something like this than doing it directly in our own protocol. regards, tom lane
On Fri, 2010-08-13 at 11:38 -0400, Tom Lane wrote: > Craig Ringer <craig@postnewspapers.com.au> writes: > > Overall: it sounds to me like SSL/TLS level compression would only be > > useful for native libpq-to-postgresql connections, and probably wouldn't > > be usable for non-libpq based database access drivers. It'd only work if > > SSL was configured, which is a limitation it'd be nice to avoid. > > I don't really see that. The case where it's sensible to use > compression on the connection is where you're pushing data across > a WAN. That's also pretty much exactly the situation where it's > sensible to use encryption. I guess there'd be a use case for > compression-without-encryption if you had a trustworthy private WAN, > but who's got one of those? > > So I'm much more in favor of doing something like this than doing it > directly in our own protocol. Well as a company who implemented compression for postgresql on the wire long ago :D. I can say it is actually useful as a whole. Even on private networks. It reduces congestion on networks, yes even 100 and 1000 networks. It provides data to the client faster because there is reduced data to push. Compression is cheap even way back then because generally the data being compressed is comparatively small. (Imagine how fast we compress with CPUs that were just two years older) Not really arguing with Tom's point as much as reminding people this has been tested and proven as a benefit. Also, let's not forget Mobile apps :D http://archives.postgresql.org/pgsql-hackers/2002-12/msg00598.php I say, let's DO IT! :) Sincerely, Joshua D. Drake -- PostgreSQL.org Major Contributor Command Prompt, Inc: http://www.commandprompt.com/ - 509.416.6579 Consulting, Training, Support, Custom Development, Engineering http://twitter.com/cmdpromptinc | http://identi.ca/commandprompt
Bruce Momjian wrote:
This is a common (and false) belief.
-- Karl
SSL is NOT always compressed at the data level. SSH is if you ask for it, but by default SSL is NOT.Craig Ringer wrote:On 13/08/2010 9:31 PM, Bruce Momjian wrote:Karl Denninger wrote:I may be blind - I don't see a way to enable this. OpenSSL "kinda" supports this - does Postgres' SSL connectivity allow it to be supported/enabled?What are you asking, exactly?As far as I can tell they're asking for transport-level compression, using gzip or similar, in much the same way as SSL/TLS currently provides transport-level encryption. Compression at the postgresql protocol level or above, so it's invisible at the level of the libpq APIs for executing statements and processing results, and doesn't change SQL processing. Since remote access is often combined with SSL, which is already supported by libpq, using SSL-integrated compression seems pretty promising if it's viable in practice. It'd avoid the pain of having to add compression to the Pg protocol by putting it "outside" the current protocol, in the SSL layer. Even better, compressing results before encrypting them makes the encrypted traffic *much* stronger against known-plaintext and pattern-based attacks. And, of course, compressing the content costs CPU time but reduces the amount of data that must then be compressed. OpenSSL does provide some transparent crypto support. See: http://www.openssl.org/docs/ssl/SSL_COMP_add_compression_method.htmlI thought all SSL traffic was compressed, unless you turned that off. It is just SSH that is always compressed?
This is a common (and false) belief.
-- Karl
Вложения
On Fri, 2010-08-13 at 11:38 -0400, Tom Lane wrote: > Craig Ringer <craig@postnewspapers.com.au> writes: > > Overall: it sounds to me like SSL/TLS level compression would only be > > useful for native libpq-to-postgresql connections, and probably wouldn't > > be usable for non-libpq based database access drivers. It'd only work if > > SSL was configured, which is a limitation it'd be nice to avoid. > > I don't really see that. The case where it's sensible to use > compression on the connection is where you're pushing data across > a WAN. That's also pretty much exactly the situation where it's > sensible to use encryption. I guess there'd be a use case for > compression-without-encryption if you had a trustworthy private WAN, > but who's got one of those? > > So I'm much more in favor of doing something like this than doing it > directly in our own protocol. Well as a company who implemented compression for postgresql on the wire long ago :D. I can say it is actually useful as a whole. Even on private networks. It reduces congestion on networks, yes even 100 and 1000 networks. It provides data to the client faster because there is reduced data to push. Compression is cheap even way back then because generally the data being compressed is comparatively small. (Imagine how fast we compress with CPUs that were just two years older) Not really arguing with Tom's point as much as reminding people this has been tested and proven as a benefit. Also, let's not forget Mobile apps :D http://archives.postgresql.org/pgsql-hackers/2002-12/msg00598.php I say, let's DO IT! :) Sincerely, Joshua D. Drake -- PostgreSQL.org Major Contributor Command Prompt, Inc: http://www.commandprompt.com/ - 509.416.6579 Consulting, Training, Support, Custom Development, Engineering http://twitter.com/cmdpromptinc | http://identi.ca/commandprompt
On Fri, Aug 13, 2010 at 10:56 AM, Joshua D. Drake <jd@commandprompt.com> wrote: > On Fri, 2010-08-13 at 11:38 -0400, Tom Lane wrote: >> >> I don't really see that. The case where it's sensible to use >> compression on the connection is where you're pushing data across >> a WAN. That's also pretty much exactly the situation where it's >> sensible to use encryption. I guess there'd be a use case for >> compression-without-encryption if you had a trustworthy private WAN, >> but who's got one of those? >> >> So I'm much more in favor of doing something like this than doing it >> directly in our own protocol. > > Well as a company who implemented compression for postgresql on the wire > long ago :D. I can say it is actually useful as a whole. Even on private > networks. We use a SSH tunnel with compression over a 2MB/s link (IPSEC VPN) from a production site to a DR site. We run slony over it and it works quite well. From what I can tell, average about 95% compression. This effectively gives us a 40MB/s link for replication. We created a Solaris SMF service to maintain the SSH tunnel and a restricted shell on the remote end. SSH connects with a private key. SMF for slony has required dependencies on the local postgresql and the SSH tunnel, so it all starts up and shutdowns down in the proper sequence. The SSH client created one forward port tunnel and one reverse port tunnel, both on localhost. To abuse the tunnel one would have to already be on the database server(s), so it seems (famous last words?) fairly secure. I think that adding compression to the SSL layer in Postgresql's wire protocol would be great. We could eliminate our kludgey tunnel (aka, yet another failure point). Just my $0.02...