Data ingestion failing when using higher Batch size

Поиск
Список
Период
Сортировка
От Rupesh Choudhary
Тема Data ingestion failing when using higher Batch size
Дата
Msg-id CAOd4mFn91NCfCevfeVjUXP0g47UcvrWkBJiJ2pK7JJuXcCmCmA@mail.gmail.com
обсуждение исходный текст
Ответы Re: Data ingestion failing when using higher Batch size  (Adrian Klaver <adrian.klaver@aklaver.com>)
Список pgsql-general
Дерево обсуждения
Hi Team ,

We have been  trying to insert data (batch inserts) using a JAVA app which uses prepared statements  with parameterized queries to insert data .We are able to insert data with small batch size  however when we increase the batch size to a higher value data ingestion fails . We are using pgbouncer for connection pooling on database side which is running in session mode  and server_reset_query set to "DISCARD ALL" . Please suggest the how can we bulk load data with higher batch size .Below the error logs that we are getting from the application end.

Below are the logs from application :-

2016-06-07T11:33:04.50-0700 [App/2]      OUT at org.postgresql.jdbc2.AbstractJdbc2Statement.executeBatch(AbstractJdbc2Statement.java:2959) ~[postgresql-9.4-1205-jdbc41.jar!/:9.4]

2016-06-07T11:33:04.50-0700 [App/2]      OUT at com.exelon.drillinginfo.service.PdenDescService.createDrillingInfo(PdenDescService.java:337) ~[app/:na]

2016-06-07T11:34:35.04-0700 [App/1]      OUT 2016-06-07 18:34:35.048  INFO 29 --- [io-64887-exec-1] c.e.d.service.PdenDescService            : Run id: 33842

2016-06-07T11:34:35.05-0700 [App/1]      OUT 2016-06-07 18:34:35.051  INFO 29 --- [io-64887-exec-1] c.e.d.service.PdenDescService            : running inserts  ********---------------------------------------------------------------***********

2016-06-07T11:34:35.05-0700 [App/1]      OUT 2016-06-07 18:34:35.058  INFO 29 --- [io-64887-exec-1] c.e.d.service.PdenDescService            : running inserts  ********---------------------------------------------------------------***********

2016-06-07T11:34:35.06-0700 [App/1]      ERR org.postgresql.util.PSQLException: ERROR: prepared statement "S_3" does not exist

2016-06-07T11:34:35.06-0700 [App/1]      ERR at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2182)

2016-06-07T11:34:35.06-0700 [App/1]      ERR at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1911)

2016-06-07T11:34:35.06-0700 [App/1]      ERR at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:338)

2016-06-07T11:34:35.06-0700 [App/1]      ERR at org.postgresql.jdbc2.AbstractJdbc2Statement.executeBatch(AbstractJdbc2Statement.java:2959)

2016-06-07T11:34:35.06-0700 [App/1]      ERR at com.exelon.drillinginfo.service.PdenDescService.createDrillingInfo(PdenDescService.java:337)

2016-06-07T11:34:35.06-0700 [App/1]      ERR at com.exelon.drillinginfo.service.PdenDescService$$FastClassBySpringCGLIB$$3080db7d.invoke(<generated>)

2016-06-07T11:34:35.06-0700 [App/1]      ERR at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)


 
Thanks & Regards,
Rupesh Choudhary

В списке pgsql-general по дате отправления:

Предыдущее
От: Edson Richter
Дата:
Сообщение: Re: Index seems "lost" after consecutive deletes
Следующее
От: Martín Marqués
Дата:
Сообщение: PgQ and pg_dump