Обсуждение: [GENERAL] Inserting millions of record in a partitioned Table

Hi guys,

with Postgresql 9.6.1 we need to insert, from a certain select query, some millions of record in a partitioned table.
The table is partitioned by day.
The datas we will import can, often, be mixed between two different days.

We noticed that if we import directly into the global table it is really, really slow.
Importing directly in the single partition is faster.

Any suggestions here please?

Thank you, best!
F

--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

On Wed, Sep 20, 2017 at 10:10 AM, Job <Job@colliniconsulting.it> wrote:

We noticed that if we import directly into the global table it is really, really slow.
Importing directly in the single partition is faster.

Do you have a rule or trigger on the main table to redirect to the partitions? You should expect that to take some extra time *per row*. Your best bet is to just import into the proper partition and make sure your application produces batch files that align with your partitions.

Either that or write a program that reads the data, determines the partition, and then inserts directly to it. It might be faster.

I wonder if this is a case of hurry up and wait.  A script which could load say 10 records, and assuming that takes much less than one second, run once per second (waiting 1000 - runtime ms) would by now have done about a million records since the question was asked.