Обсуждение: Invalid memory allocation error with pg_recvlogical or with libPQ logical connection
Invalid memory allocation error with pg_recvlogical or with libPQ logical connection
От
jitesh tiwari
Дата:
Hi all,
I have a use case in postgresql where I have inserted 1 row in a table with 3 columns as described below.
create table xyz (
id int not null primary key,
col1 bytea,
col2 text
);
The table has 1 row with approx 700 MB of data. The bytea column data in the row has around 500MB of data while the text column has data of around 150MB approximately.
When I try to fetch the rows with the logical decoding using the test_decoding plugin with libPQ connection, I get the below error -
ERROR: invalid memory alloc request size 1073741825
A similar error I see with pg_recvlogical tool.
pg_recvlogical: error: unexpected termination of replication stream: ERROR:
invalid memory alloc request size 1073741825
CONTEXT: slot "elob5121_a818598eea3b7a71", output plugin "test_decoding", in
the change callback, associated LSN 1/BC5681E0
pg_recvlogical: disconnected; waiting 5 seconds to try again.
I have enabled the Server to debug log and I see a similar error -
2023-03-01 07:50:05.769 UTC [33486] DETAIL: There are no running
transactions.
2023-03-01 07:50:05.769 UTC [33486] STATEMENT: START_REPLICATION SLOT
"elob5121_a818598eea3b7a71" LOGICAL 1/9A163D88 ("include-timestamp",
"include-xids")
2023-03-01 07:50:08.122 UTC [33486] ERROR: invalid memory alloc request size
1073741825
Please suggest if it is a known issue or limitation in postgresql backend code. If so please point to the documentation link for the same. If there is any workaround as well please update me.
I have a use case in postgresql where I have inserted 1 row in a table with 3 columns as described below.
create table xyz (
id int not null primary key,
col1 bytea,
col2 text
);
The table has 1 row with approx 700 MB of data. The bytea column data in the row has around 500MB of data while the text column has data of around 150MB approximately.
When I try to fetch the rows with the logical decoding using the test_decoding plugin with libPQ connection, I get the below error -
ERROR: invalid memory alloc request size 1073741825
A similar error I see with pg_recvlogical tool.
pg_recvlogical: error: unexpected termination of replication stream: ERROR:
invalid memory alloc request size 1073741825
CONTEXT: slot "elob5121_a818598eea3b7a71", output plugin "test_decoding", in
the change callback, associated LSN 1/BC5681E0
pg_recvlogical: disconnected; waiting 5 seconds to try again.
I have enabled the Server to debug log and I see a similar error -
2023-03-01 07:50:05.769 UTC [33486] DETAIL: There are no running
transactions.
2023-03-01 07:50:05.769 UTC [33486] STATEMENT: START_REPLICATION SLOT
"elob5121_a818598eea3b7a71" LOGICAL 1/9A163D88 ("include-timestamp",
"include-xids")
2023-03-01 07:50:08.122 UTC [33486] ERROR: invalid memory alloc request size
1073741825
Please suggest if it is a known issue or limitation in postgresql backend code. If so please point to the documentation link for the same. If there is any workaround as well please update me.
Regards,
Jitesh Kumar
Hello Jitesh Kumar,
without having a deeper look, it looks like there is something exceeding our 1 GB limit (after encoding).
I usually try to stay below 500 MB, so I can safely base16 encode my data. That makes administration way easier in general.
When I need to store larger bytea, I usually split it up into multiple parts and store them into separate rows.
I assume without further investigation, that this is related to the 1 GB field size, documented here: https://www.postgresql.org/docs/current/limits.html
I understand this documentation is probably not very helpful to you. Maybe we can improve on the docs somewhere.
Regards
Arne
From: jitesh tiwari <jitesh120@gmail.com>
Sent: Monday, March 13, 2023 08:01
To: pgsql-bugs@postgresql.org <pgsql-bugs@postgresql.org>
Subject: Invalid memory allocation error with pg_recvlogical or with libPQ logical connection
Sent: Monday, March 13, 2023 08:01
To: pgsql-bugs@postgresql.org <pgsql-bugs@postgresql.org>
Subject: Invalid memory allocation error with pg_recvlogical or with libPQ logical connection
Hi all,
I have a use case in postgresql where I have inserted 1 row in a table with 3 columns as described below.
create table xyz (
id int not null primary key,
col1 bytea,
col2 text
);
The table has 1 row with approx 700 MB of data. The bytea column data in the row has around 500MB of data while the text column has data of around 150MB approximately.
When I try to fetch the rows with the logical decoding using the test_decoding plugin with libPQ connection, I get the below error -
ERROR: invalid memory alloc request size 1073741825
A similar error I see with pg_recvlogical tool.
pg_recvlogical: error: unexpected termination of replication stream: ERROR:
invalid memory alloc request size 1073741825
CONTEXT: slot "elob5121_a818598eea3b7a71", output plugin "test_decoding", in
the change callback, associated LSN 1/BC5681E0
pg_recvlogical: disconnected; waiting 5 seconds to try again.
I have enabled the Server to debug log and I see a similar error -
2023-03-01 07:50:05.769 UTC [33486] DETAIL: There are no running
transactions.
2023-03-01 07:50:05.769 UTC [33486] STATEMENT: START_REPLICATION SLOT
"elob5121_a818598eea3b7a71" LOGICAL 1/9A163D88 ("include-timestamp",
"include-xids")
2023-03-01 07:50:08.122 UTC [33486] ERROR: invalid memory alloc request size
1073741825
Please suggest if it is a known issue or limitation in postgresql backend code. If so please point to the documentation link for the same. If there is any workaround as well please update me.
I have a use case in postgresql where I have inserted 1 row in a table with 3 columns as described below.
create table xyz (
id int not null primary key,
col1 bytea,
col2 text
);
The table has 1 row with approx 700 MB of data. The bytea column data in the row has around 500MB of data while the text column has data of around 150MB approximately.
When I try to fetch the rows with the logical decoding using the test_decoding plugin with libPQ connection, I get the below error -
ERROR: invalid memory alloc request size 1073741825
A similar error I see with pg_recvlogical tool.
pg_recvlogical: error: unexpected termination of replication stream: ERROR:
invalid memory alloc request size 1073741825
CONTEXT: slot "elob5121_a818598eea3b7a71", output plugin "test_decoding", in
the change callback, associated LSN 1/BC5681E0
pg_recvlogical: disconnected; waiting 5 seconds to try again.
I have enabled the Server to debug log and I see a similar error -
2023-03-01 07:50:05.769 UTC [33486] DETAIL: There are no running
transactions.
2023-03-01 07:50:05.769 UTC [33486] STATEMENT: START_REPLICATION SLOT
"elob5121_a818598eea3b7a71" LOGICAL 1/9A163D88 ("include-timestamp",
"include-xids")
2023-03-01 07:50:08.122 UTC [33486] ERROR: invalid memory alloc request size
1073741825
Please suggest if it is a known issue or limitation in postgresql backend code. If so please point to the documentation link for the same. If there is any workaround as well please update me.
Regards,
Jitesh Kumar
Re: Invalid memory allocation error with pg_recvlogical or with libPQ logical connection
От
jitesh tiwari
Дата:
Thanks, I'll take a look.
Does that mean that bytea data cannot be stored in the same row of of data size >=500 MB? If that is the case we should document this clearly for the end user.
Regards,
Jitesh
On Mon, Mar 13, 2023 at 5:13 PM Arne Roland <A.Roland@index.de> wrote:
Hello Jitesh Kumar,without having a deeper look, it looks like there is something exceeding our 1 GB limit (after encoding).I usually try to stay below 500 MB, so I can safely base16 encode my data. That makes administration way easier in general.When I need to store larger bytea, I usually split it up into multiple parts and store them into separate rows.I assume without further investigation, that this is related to the 1 GB field size, documented here: https://www.postgresql.org/docs/current/limits.htmlI understand this documentation is probably not very helpful to you. Maybe we can improve on the docs somewhere.RegardsArneFrom: jitesh tiwari <jitesh120@gmail.com>
Sent: Monday, March 13, 2023 08:01
To: pgsql-bugs@postgresql.org <pgsql-bugs@postgresql.org>
Subject: Invalid memory allocation error with pg_recvlogical or with libPQ logical connectionHi all,
I have a use case in postgresql where I have inserted 1 row in a table with 3 columns as described below.
create table xyz (
id int not null primary key,
col1 bytea,
col2 text
);
The table has 1 row with approx 700 MB of data. The bytea column data in the row has around 500MB of data while the text column has data of around 150MB approximately.
When I try to fetch the rows with the logical decoding using the test_decoding plugin with libPQ connection, I get the below error -
ERROR: invalid memory alloc request size 1073741825
A similar error I see with pg_recvlogical tool.
pg_recvlogical: error: unexpected termination of replication stream: ERROR:
invalid memory alloc request size 1073741825
CONTEXT: slot "elob5121_a818598eea3b7a71", output plugin "test_decoding", in
the change callback, associated LSN 1/BC5681E0
pg_recvlogical: disconnected; waiting 5 seconds to try again.
I have enabled the Server to debug log and I see a similar error -
2023-03-01 07:50:05.769 UTC [33486] DETAIL: There are no running
transactions.
2023-03-01 07:50:05.769 UTC [33486] STATEMENT: START_REPLICATION SLOT
"elob5121_a818598eea3b7a71" LOGICAL 1/9A163D88 ("include-timestamp",
"include-xids")
2023-03-01 07:50:08.122 UTC [33486] ERROR: invalid memory alloc request size
1073741825
Please suggest if it is a known issue or limitation in postgresql backend code. If so please point to the documentation link for the same. If there is any workaround as well please update me.Regards,Jitesh Kumar