Обсуждение: Query out of memory
Hi,
I am running the below query. Table has 21 million records. I get an Out Of Memory error after a while.(from both pgadmin and psql). Can someone review DB parameters given below.
select t.*,g.column,a.column from
gk_staging g, transaction t,account a
where
g.accountcodeis not null AND
g.accountcode::text <> '' AND
length(g.accountcode)=13 AND
g.closeid::text=t.transactionid::text AND
subsrting(g.accountcode::text,8)=a.mask_code::text
Below are system parameters.
shared_buffers=3GB
work_mem=2GB
effective_cache_size=10GB
maintenance_work_mem=1GB
max_connections=250
I am unable to paste explain plan here due to security concerns.
Regards,
Aditya.
Database has 20GB RAM.
On Mon, Oct 18, 2021 at 10:12 PM aditya desai <admad123@gmail.com> wrote:
Hi,I am running the below query. Table has 21 million records. I get an Out Of Memory error after a while.(from both pgadmin and psql). Can someone review DB parameters given below.select t.*,g.column,a.column fromgk_staging g, transaction t,account awhereg.accountcodeis not null ANDg.accountcode::text <> '' ANDlength(g.accountcode)=13 ANDg.closeid::text=t.transactionid::text ANDsubsrting(g.accountcode::text,8)=a.mask_code::textBelow are system parameters.shared_buffers=3GBwork_mem=2GBeffective_cache_size=10GBmaintenance_work_mem=1GBmax_connections=250I am unable to paste explain plan here due to security concerns.Regards,Aditya.
Work memory 2 GB ?
Is this intentional?
How many max active connections do you see?
if you have too many connections. You can try toning it down to
https://pgtune.leopard.in.ua/ to start with.
On Mon, Oct 18, 2021, 10:13 PM aditya desai <admad123@gmail.com> wrote:
Hi,I am running the below query. Table has 21 million records. I get an Out Of Memory error after a while.(from both pgadmin and psql). Can someone review DB parameters given below.select t.*,g.column,a.column fromgk_staging g, transaction t,account awhereg.accountcodeis not null ANDg.accountcode::text <> '' ANDlength(g.accountcode)=13 ANDg.closeid::text=t.transactionid::text ANDsubsrting(g.accountcode::text,8)=a.mask_code::textBelow are system parameters.shared_buffers=3GBwork_mem=2GBeffective_cache_size=10GBmaintenance_work_mem=1GBmax_connections=250I am unable to paste explain plan here due to security concerns.Regards,Aditya.
Not many active connections. Only 30-40.
On Monday, October 18, 2021, Vijaykumar Jain <vijaykumarjain.github@gmail.com> wrote:
On Monday, October 18, 2021, Vijaykumar Jain <vijaykumarjain.github@gmail.com> wrote:
Work memory 2 GB ?Is this intentional?How many max active connections do you see?if you have too many connections. You can try toning it down tohttps://pgtune.leopard.in.ua/ to start with.On Mon, Oct 18, 2021, 10:13 PM aditya desai <admad123@gmail.com> wrote:Hi,I am running the below query. Table has 21 million records. I get an Out Of Memory error after a while.(from both pgadmin and psql). Can someone review DB parameters given below.select t.*,g.column,a.column fromgk_staging g, transaction t,account awhereg.accountcodeis not null ANDg.accountcode::text <> '' ANDlength(g.accountcode)=13 ANDg.closeid::text=t.transactionid::text AND subsrting(g.accountcode::text,8)=a.mask_code::text Below are system parameters.shared_buffers=3GBwork_mem=2GBeffective_cache_size=10GBmaintenance_work_mem=1GBmax_connections=250I am unable to paste explain plan here due to security concerns.Regards,Aditya.
Le lun. 18 oct. 2021 à 21:03, aditya desai <admad123@gmail.com> a écrit :
Not many active connections. Only 30-40.
This means you can consume up to 60-80 GB. Way much more than the available RAM. You should lower your work_mem value.
On Monday, October 18, 2021, Vijaykumar Jain <vijaykumarjain.github@gmail.com> wrote:Work memory 2 GB ?Is this intentional?How many max active connections do you see?if you have too many connections. You can try toning it down tohttps://pgtune.leopard.in.ua/ to start with.On Mon, Oct 18, 2021, 10:13 PM aditya desai <admad123@gmail.com> wrote:Hi,I am running the below query. Table has 21 million records. I get an Out Of Memory error after a while.(from both pgadmin and psql). Can someone review DB parameters given below.select t.*,g.column,a.column fromgk_staging g, transaction t,account awhereg.accountcodeis not null ANDg.accountcode::text <> '' ANDlength(g.accountcode)=13 ANDg.closeid::text=t.transactionid::text ANDsubsrting(g.accountcode::text,8)=a.mask_code::textBelow are system parameters.shared_buffers=3GBwork_mem=2GBeffective_cache_size=10GBmaintenance_work_mem=1GBmax_connections=250I am unable to paste explain plan here due to security concerns.Regards,Aditya.
This is how I received a query from the App Team. They have migrated from Oracle to Postgres. I see in Oracle where it is working fine and also has the same joins. Some of the developers increased the work_mem. I will try and tone it down. Will get back to you. Thanks.
On Tue, Oct 19, 2021 at 1:28 AM Geri Wright <geri.w123@gmail.com> wrote:
Hi,It looks like you have Cartesian joins in the query. Try updating your where clause to includeAnd g.columnname = t.columnnameAnd t.columnname2 = a.columnname2On Mon, Oct 18, 2021, 12:43 PM aditya desai <admad123@gmail.com> wrote:Hi,I am running the below query. Table has 21 million records. I get an Out Of Memory error after a while.(from both pgadmin and psql). Can someone review DB parameters given below.select t.*,g.column,a.column fromgk_staging g, transaction t,account awhereg.accountcodeis not null ANDg.accountcode::text <> '' ANDlength(g.accountcode)=13 ANDg.closeid::text=t.transactionid::text ANDsubsrting(g.accountcode::text,8)=a.mask_code::textBelow are system parameters.shared_buffers=3GBwork_mem=2GBeffective_cache_size=10GBmaintenance_work_mem=1GBmax_connections=250I am unable to paste explain plan here due to security concerns.Regards,Aditya.