Re: Invalid memory allocation error with pg_recvlogical or with libPQ logical connection

Hello Jitesh Kumar,

without having a deeper look, it looks like there is something exceeding our 1 GB limit (after encoding).

I usually try to stay below 500 MB, so I can safely base16 encode my data. That makes administration way easier in general.

When I need to store larger bytea, I usually split it up into multiple parts and store them into separate rows.

I assume without further investigation, that this is related to the 1 GB field size, documented here: https://www.postgresql.org/docs/current/limits.html

I understand this documentation is probably not very helpful to you. Maybe we can improve on the docs somewhere.

Regards

Arne

---

From: jitesh tiwari <jitesh120@gmail.com>
Sent: Monday, March 13, 2023 08:01
To: pgsql-bugs@postgresql.org <pgsql-bugs@postgresql.org>
Subject: Invalid memory allocation error with pg_recvlogical or with libPQ logical connection

 

 Hi all,

I have a use case in postgresql where I have inserted 1 row in a table with 3 columns as described below.
create table xyz (
id int not null primary key,
col1 bytea,
col2 text
);

The table has 1 row with approx 700 MB of data. The bytea column data in the row has around 500MB of data while the text column has data of around 150MB approximately.
When I try to fetch the rows with the logical decoding using the test_decoding plugin with libPQ connection, I get the below error -

ERROR: invalid memory alloc request size 1073741825

A similar error I see with pg_recvlogical tool.
pg_recvlogical: error: unexpected termination of replication stream: ERROR:  
invalid memory alloc request size 1073741825
CONTEXT:  slot "elob5121_a818598eea3b7a71", output plugin "test_decoding", in
the change callback, associated LSN 1/BC5681E0
pg_recvlogical: disconnected; waiting 5 seconds to try again.

I have enabled the Server to debug log and I see a similar error -
2023-03-01 07:50:05.769 UTC [33486] DETAIL:  There are no running
transactions.
2023-03-01 07:50:05.769 UTC [33486] STATEMENT:  START_REPLICATION SLOT
"elob5121_a818598eea3b7a71" LOGICAL 1/9A163D88 ("include-timestamp",
"include-xids")
2023-03-01 07:50:08.122 UTC [33486] ERROR:  invalid memory alloc request size
1073741825

Please suggest if it is a known issue or limitation in postgresql backend code. If so please point to the documentation link for the same. If there is any workaround as well please update me.

Regards, 

Jitesh Kumar