question regarding copyData containers

Поиск
Список
Период
Сортировка
От Jerome Wagner
Тема question regarding copyData containers
Дата
Msg-id CA+=V_fNc60NLg9SmoL0mY1EkMsnqcJ0w0hQG2Ye6SWLVLdbrvg@mail.gmail.com
обсуждение исходный текст
Ответы Re: question regarding copyData containers  (Tom Lane <tgl@sss.pgh.pa.us>)
Re: question regarding copyData containers  (Andres Freund <andres@anarazel.de>)
Список pgsql-hackers
Hello,

I have been working on a node.js streaming client for different COPY scenarios.
usually, during CopyOut, clients tend to buffer network chunks until they have gathered a full copyData message and pass that to the user.

In some cases, this can lead to very large copyData messages. when there are very long text fields or bytea fields it will require a lot of memory to be handled (up to 1GB I think in the worst case scenario)

In COPY TO, I managed to relax that requirement, considering that copyData is simply a transparent container. For each network chunk, the relevent message content is forwarded which makes for 64KB chunks at most.

If that makes things clearer, here is an example scenarios, with 4 network chunks received and the way they are forwarded to the client.

in: CopyData Int32Len Byten1
in: Byten2
in: Byten3
in: CopyData Int32Len Byten4

out: Byten1
out: Byten2
out: Byten3  
out: Byten4

We loose the semantics of the "row" that copyData has according to the documentation 
>The backend sends a CopyOutResponse message to the frontend, followed by zero or more >CopyData messages (**always one per row**), followed by CopyDone

but it is not a problem because the raw bytes are still parsable (rows + fields) in text mode (tsv) and in binary mode)

Now I started working on copyBoth and logical decoding scenarios. In this case, the server send series of copyData. 1 copyData containing 1 message :

at the network chunk level, in the case of large fields, we can observe

in: CopyData Int32 XLogData Int64  Int64  Int64 Byten1
in: Byten2 
in: CopyData Int32 XLogData Int64  Int64  Int64 Byten3
in: CopyData Int32 XLogData Int64  Int64  Int64 Byten4

out: XLogData Int64  Int64  Int64 Byten1
out: Byten2
out: XLogData Int64  Int64  Int64 Byten3
out: XLogData Int64  Int64  Int64 Byten4

but at the XLogData level, the protocol is not self-describing its length, so there is no real way of knowing where the first XLogData ends apart from
 - knowing the length of the first copyData (4 + 1 + 3*8 + n1 + n2)
 - knowing the internals of the output plugin and benefit from a plugin that self-describe its span 

when a network chunks contains several copyDatas
in: CopyData Int32 XLogData Int64  Int64  Int64 Byten1  CopyData Int32 XLogData Int64  Int64  Int64 Byten2  
we have
out: XLogData Int64  Int64  Int64 Byten1 XLogData Int64  Int64  Int64 Byten2

and with test_decoding for example it is impossible to know where the test_decoding output ends without remembering the original length of the copyData.

now my question is the following : 
is it ok to consider that over the long term copyData is simply a transport container that exists only to allow the multiplexing of events in the protocol but that messages inside could be chunked over several copyData events ?

if we put test_decoding apart, do you consider that output plugins XLogData should be self-aware of their length ? I suppose (but did not fully verify yet) that this is the case for pgoutput ? I suppose that wal2json could also be parsed by balancing the brackets.

I am wondering because when a client sends copyData to the server, the documentation says
>The message boundaries are not required to have anything to do with row boundaries, >although that is often a reasonable choice.  

I hope that my message will ring a bell on the list.
I tried the best I could to describe my very specific research.
Thank you for your help,
---
Jérôme





В списке pgsql-hackers по дате отправления:

Предыдущее
От: Alvaro Herrera
Дата:
Сообщение: Re: Towards easier AMs: Cleaning up inappropriate use of name"relkind"
Следующее
От: Andres Freund
Дата:
Сообщение: Re: Why is pq_begintypsend so slow?