Re: libpq compression

On 22.06.2018 20:56, Robbie Harwood wrote:
> Konstantin Knizhnik <k.knizhnik@postgrespro.ru> writes:
>
>> On 22.06.2018 18:59, Robbie Harwood wrote:
>>> Konstantin Knizhnik <k.knizhnik@postgrespro.ru> writes:
>>>> On 21.06.2018 20:14, Robbie Harwood wrote:
>>>>> Konstantin Knizhnik <k.knizhnik@postgrespro.ru> writes:
>>>>>> On 21.06.2018 17:56, Robbie Harwood wrote:
>>>>>>> Konstantin Knizhnik <k.knizhnik@postgrespro.ru> writes:
>>>>>>>> On 20.06.2018 23:34, Robbie Harwood wrote:
>>>>>>>>> Konstantin Knizhnik <k.knizhnik@postgrespro.ru> writes:
>>>>>>>>>
>>>>>>>>>> My idea was the following: client want to use compression. But
>>>>>>>>>> server may reject this attempt (for any reasons: it doesn't
>>>>>>>>>> support it, has no proper compression library, do not want to
>>>>>>>>>> spend CPU for decompression,...) Right now compression
>>>>>>>>>> algorithm is hardcoded. But in future client and server may
>>>>>>>>>> negotiate to choose proper compression protocol.  This is why
>>>>>>>>>> I prefer to perform negotiation between client and server to
>>>>>>>>>> enable compression.
>>>>>>>>> Well, for negotiation you could put the name of the algorithm
>>>>>>>>> you want in the startup.  It doesn't have to be a boolean for
>>>>>>>>> compression, and then you don't need an additional round-trip.
>>>>>>>> Sorry, I can only repeat arguments I already mentioned:
>>>>>>>>
>>>>>>>> - in future it may be possible to specify compression algorithm
>>>>>>>>
>>>>>>>> - even with boolean compression option server may have some
>>>>>>>> reasons to reject client's request to use compression
>>>>>>>>
>>>>>>>> Extra flexibility is always good thing if it doesn't cost too
>>>>>>>> much. And extra round of negotiation in case of enabling
>>>>>>>> compression seems to me not to be a high price for it.
>>>>>>> You already have this flexibility even without negotiation.  I
>>>>>>> don't want you to lose your flexibility.  Protocol looks like
>>>>>>> this:
>>>>>>>
>>>>>>> - Client sends connection option "compression" with list of
>>>>>>>      algorithms it wants to use (comma-separated, or something).
>>>>>>>
>>>>>>> - First packet that the server can compress one of those algorithms
>>>>>>>      (or none, if it doesn't want to turn on compression).
>>>>>>>
>>>>>>> No additional round-trips needed.
>>>>>> This is exactly how it works now...  Client includes compression
>>>>>> option in connection string and server replies with special
>>>>>> message ('Z') if it accepts request to compress traffic between
>>>>>> this client and server.
>>>>> No, it's not.  You don't need this message.  If the server receives
>>>>> a compression request, it should just turn compression on (or not),
>>>>> and then have the client figure out whether it got compression
>>>>> back.
>>>> How it will managed to do it. It receives some reply and first of
>>>> all it should know whether it has to be decompressed or not.
>>> You can tell whether a message is compressed by looking at it.  The
>>> way the protocol works, every message has a type associated with it:
>>> a single byte, like 'R', that says what kind of message it is.
>> Compressed message can contain any sequence of bytes, including 'R':)
> Then tag your messages with a type byte.  Or do it the other way around
> - look for the zstd framing within a message.
>
> Please, try to work with me on this instead of fighting every design
> change.
Sorry, I do not want fighting.
I am always vote for peace and constructive dialog.

But it is hard to me to understand and accept your arguments.

I do not understand why secure_read function is better place for 
handling compression than pq_recvbuf.
And why it is destroying existed model.
I already mentioned my arguments:
1. I want to use the same code for frontend and backend.
2. I think that streaming compression can be used not only for libpq. 
This is why I tried to make zpq_stream independent from
communication layer and pass here callbacks for sending/receiving data.

If pq_recvbuf is not right place for performing decommpression, I can 
introduce some other function,  like read_raw or something like that and 
do decompression here. But I do not see much sense in it.


Concerning necessity of special message for acknowledging compression by 
server: I also do not understand why you do not like idea to send some 
message and what is wrong with it. What you are suggesting "then tag 
your message" actually is the same as sending new message.
Because what is the difference between tag 'Z' and message with code 'Z'?

Sorry, but  I do not understand problems you are going to solve and do 
not see any arguments except "I can not accept it".

> Thanks,
> --Robbie